Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 19:55:58.209506 5677 test_util.cc:268] Using random seed: -536026671
I20240501 19:55:58.215860 5677 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:55:58.215978 5677 ts_itest-base.cc:113] --------------
I20240501 19:55:58.216012 5677 ts_itest-base.cc:114] 6 tablet servers
I20240501 19:55:58.216035 5677 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:55:58.216051 5677 ts_itest-base.cc:116] --------------
2024-05-01T19:55:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:55:58Z Disabled control of system clock
I20240501 19:55:58.224858 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.139.126:35927
--webserver_interface=127.5.139.126
--webserver_port=0
--builtin_ntp_servers=127.5.139.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.139.126:35927
--location_mapping_cmd=/tmp/dist-test-taskuLQw07/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 19:55:58.325950 5689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.326174 5689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.326215 5689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.329203 5689 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.329262 5689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:55:58.329283 5689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.329310 5689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:55:58.329327 5689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:55:58.332803 5689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskuLQw07/build/debug/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.5.139.126:35927
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.139.126:35927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.139.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.5689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.333644 5689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.334700 5689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.342656 5695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.343269 5694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.344045 5689 server_base.cc:1034] running on GCE node
W20240501 19:55:58.344637 5697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.345152 5689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.346002 5689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.347151 5689 hybrid_clock.cc:648] HybridClock initialized: now 1714593358347141 us; error 26 us; skew 500 ppm
I20240501 19:55:58.356184 5689 webserver.cc:460] Webserver started at http://127.5.139.126:33163/ using document root <none> and password file <none>
I20240501 19:55:58.356724 5689 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.356786 5689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.356972 5689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.358587 5689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4a381a255c7d46878094e496e9d7ef5a"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.359045 5689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4a381a255c7d46878094e496e9d7ef5a"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.362207 5689 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:58.364238 5703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.365211 5689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:55:58.365319 5689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4a381a255c7d46878094e496e9d7ef5a"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.365401 5689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.391080 5689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.391745 5689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.391919 5689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.398092 5689 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.126:35927
I20240501 19:55:58.398154 5756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.126:35927 every 8 connection(s)
I20240501 19:55:58.399035 5689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:55:58.400122 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 5689
I20240501 19:55:58.400266 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:55:58.402096 5757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:55:58.407626 5757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a: Bootstrap starting.
I20240501 19:55:58.410127 5757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a: Neither blocks nor log segments found. Creating new log.
I20240501 19:55:58.410923 5757 log.cc:824] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a: Log is configured to *not* fsync() on all Append() calls
I20240501 19:55:58.412775 5757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a: No bootstrap required, opened a new log
I20240501 19:55:58.415756 5757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a381a255c7d46878094e496e9d7ef5a" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35927 } }
I20240501 19:55:58.415993 5757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:55:58.416112 5757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a381a255c7d46878094e496e9d7ef5a, State: Initialized, Role: FOLLOWER
I20240501 19:55:58.416549 5757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a381a255c7d46878094e496e9d7ef5a" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35927 } }
I20240501 19:55:58.416668 5757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:55:58.416731 5757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:55:58.416927 5757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:55:58.417722 5757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a381a255c7d46878094e496e9d7ef5a" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35927 } }
I20240501 19:55:58.418032 5757 leader_election.cc:302] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a381a255c7d46878094e496e9d7ef5a; no voters:
I20240501 19:55:58.418282 5757 leader_election.cc:288] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:55:58.418393 5762 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:55:58.418584 5762 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [term 1 LEADER]: Becoming Leader. State: Replica: 4a381a255c7d46878094e496e9d7ef5a, State: Running, Role: LEADER
I20240501 19:55:58.418920 5762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a381a255c7d46878094e496e9d7ef5a" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35927 } }
I20240501 19:55:58.419273 5757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:55:58.420640 5764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a381a255c7d46878094e496e9d7ef5a. Latest consensus state: current_term: 1 leader_uuid: "4a381a255c7d46878094e496e9d7ef5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a381a255c7d46878094e496e9d7ef5a" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35927 } } }
I20240501 19:55:58.420755 5764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:58.421118 5772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:55:58.421087 5763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a381a255c7d46878094e496e9d7ef5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a381a255c7d46878094e496e9d7ef5a" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35927 } } }
I20240501 19:55:58.421171 5763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a [sys.catalog]: This master's current role is: LEADER
I20240501 19:55:58.424333 5772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:55:58.429103 5772 catalog_manager.cc:1347] Generated new cluster ID: 1cd21fabacd34d00b69c2048da0fefb9
I20240501 19:55:58.429163 5772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:55:58.436441 5772 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:55:58.437284 5772 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:55:58.446282 5772 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 4a381a255c7d46878094e496e9d7ef5a: Generated new TSK 0
I20240501 19:55:58.446910 5772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:55:58.457772 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.65:0
--local_ip_for_outbound_sockets=127.5.139.65
--webserver_interface=127.5.139.65
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--builtin_ntp_servers=127.5.139.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:55:58.560832 5781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.561029 5781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.561123 5781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.564066 5781 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.564136 5781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.564231 5781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.65
I20240501 19:55:58.567734 5781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.139.65
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--never_fsync=true
--heap_profile_path=/tmp/kudu.5781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.65
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.568624 5781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.569772 5781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.578323 5786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.579010 5789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.579703 5787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.580404 5781 server_base.cc:1034] running on GCE node
I20240501 19:55:58.580786 5781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.581416 5781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.582577 5781 hybrid_clock.cc:648] HybridClock initialized: now 1714593358582554 us; error 45 us; skew 500 ppm
I20240501 19:55:58.584594 5781 webserver.cc:460] Webserver started at http://127.5.139.65:42893/ using document root <none> and password file <none>
I20240501 19:55:58.585314 5781 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.585387 5781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.585610 5781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.587239 5781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4629e756401746a8bf4652f995bddb44"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.587718 5781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4629e756401746a8bf4652f995bddb44"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.590963 5781 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:55:58.593083 5795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.594053 5781 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.594167 5781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4629e756401746a8bf4652f995bddb44"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.594250 5781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.603863 5781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.604398 5781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.604555 5781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.605088 5781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.605898 5781 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.605947 5781 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.606004 5781 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.606035 5781 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.614104 5781 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.65:40845
I20240501 19:55:58.614154 5908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.65:40845 every 8 connection(s)
I20240501 19:55:58.615016 5781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:55:58.623215 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 5781
I20240501 19:55:58.623332 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:55:58.623526 5909 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35927
I20240501 19:55:58.623740 5909 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.624313 5909 heartbeater.cc:511] Master 127.5.139.126:35927 requested a full tablet report, sending...
I20240501 19:55:58.625566 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.66:0
--local_ip_for_outbound_sockets=127.5.139.66
--webserver_interface=127.5.139.66
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--builtin_ntp_servers=127.5.139.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:55:58.675729 5720 ts_manager.cc:194] Registered new tserver with Master: 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845)
I20240501 19:55:58.677279 5720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.65:53845
W20240501 19:55:58.747504 5912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.747717 5912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.747761 5912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.750851 5912 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.750922 5912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.751013 5912 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.66
I20240501 19:55:58.754642 5912 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.139.66
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--never_fsync=true
--heap_profile_path=/tmp/kudu.5912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.66
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.755538 5912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.756645 5912 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.765117 5918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.765724 5921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.766443 5919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.767202 5912 server_base.cc:1034] running on GCE node
I20240501 19:55:58.767571 5912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.768142 5912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.769284 5912 hybrid_clock.cc:648] HybridClock initialized: now 1714593358769266 us; error 32 us; skew 500 ppm
I20240501 19:55:58.771219 5912 webserver.cc:460] Webserver started at http://127.5.139.66:44379/ using document root <none> and password file <none>
I20240501 19:55:58.771760 5912 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.771816 5912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.771989 5912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.773638 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "3489b69a0428491e957e46effb9b1146"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.774097 5912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "3489b69a0428491e957e46effb9b1146"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.777443 5912 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:55:58.779474 5927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.780409 5912 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:58.780510 5912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "3489b69a0428491e957e46effb9b1146"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.780591 5912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.807282 5912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.807946 5912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.808109 5912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.808658 5912 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.809574 5912 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.809621 5912 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.809664 5912 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.809710 5912 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.818222 5912 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.66:33835
I20240501 19:55:58.818269 6040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.66:33835 every 8 connection(s)
I20240501 19:55:58.819231 5912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:55:58.823819 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 5912
I20240501 19:55:58.823920 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:55:58.826057 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.67:0
--local_ip_for_outbound_sockets=127.5.139.67
--webserver_interface=127.5.139.67
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--builtin_ntp_servers=127.5.139.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:55:58.827983 6041 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35927
I20240501 19:55:58.828204 6041 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.828670 6041 heartbeater.cc:511] Master 127.5.139.126:35927 requested a full tablet report, sending...
I20240501 19:55:58.860684 5720 ts_manager.cc:194] Registered new tserver with Master: 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:55:58.861459 5720 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.66:46861
W20240501 19:55:58.931527 6044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:58.931727 6044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:58.931795 6044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:58.934767 6044 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:58.934921 6044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:58.935058 6044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.67
I20240501 19:55:58.938639 6044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.139.67
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--never_fsync=true
--heap_profile_path=/tmp/kudu.6044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.67
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:58.939442 6044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:58.940546 6044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:58.948464 6050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.949116 6051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:58.949803 6053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:58.950507 6044 server_base.cc:1034] running on GCE node
I20240501 19:55:58.950826 6044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:58.951337 6044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:58.952510 6044 hybrid_clock.cc:648] HybridClock initialized: now 1714593358952498 us; error 24 us; skew 500 ppm
I20240501 19:55:58.954392 6044 webserver.cc:460] Webserver started at http://127.5.139.67:43191/ using document root <none> and password file <none>
I20240501 19:55:58.954916 6044 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:58.954974 6044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:58.955135 6044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:58.956645 6044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.957141 6044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.960211 6044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 19:55:58.962347 6059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.963301 6044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:55:58.963413 6044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
format_stamp: "Formatted at 2024-05-01 19:55:58 on dist-test-slave-twc7"
I20240501 19:55:58.963492 6044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:58.973520 6044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:58.974035 6044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:58.974169 6044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:58.974614 6044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:58.975455 6044 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:58.975502 6044 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.975553 6044 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:58.975580 6044 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:58.983458 6044 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.67:36087
I20240501 19:55:58.983500 6172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.67:36087 every 8 connection(s)
I20240501 19:55:58.984256 6044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:55:58.991947 6173 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35927
I20240501 19:55:58.992147 6173 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:58.992408 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 6044
I20240501 19:55:58.992491 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:55:58.992660 6173 heartbeater.cc:511] Master 127.5.139.126:35927 requested a full tablet report, sending...
I20240501 19:55:58.994345 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.68:0
--local_ip_for_outbound_sockets=127.5.139.68
--webserver_interface=127.5.139.68
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--builtin_ntp_servers=127.5.139.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:55:59.028888 5719 ts_manager.cc:194] Registered new tserver with Master: d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:55:59.029851 5719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.67:32939
W20240501 19:55:59.100106 6177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.100302 6177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.100342 6177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.103219 6177 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:59.103276 6177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.103338 6177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.68
I20240501 19:55:59.108071 6177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.139.68
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--never_fsync=true
--heap_profile_path=/tmp/kudu.6177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.68
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.108857 6177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.109881 6177 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.118074 6183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.118714 6185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.119436 6182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.120173 6177 server_base.cc:1034] running on GCE node
I20240501 19:55:59.120524 6177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.121167 6177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.122341 6177 hybrid_clock.cc:648] HybridClock initialized: now 1714593359122326 us; error 27 us; skew 500 ppm
I20240501 19:55:59.124119 6177 webserver.cc:460] Webserver started at http://127.5.139.68:35407/ using document root <none> and password file <none>
I20240501 19:55:59.124631 6177 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.124694 6177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.124871 6177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.126477 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "fb1af421bf6a480587c69697af9f39dd"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-twc7"
I20240501 19:55:59.126940 6177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "fb1af421bf6a480587c69697af9f39dd"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-twc7"
I20240501 19:55:59.130052 6177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:59.132010 6191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.132913 6177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.133013 6177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "fb1af421bf6a480587c69697af9f39dd"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-twc7"
I20240501 19:55:59.133127 6177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.155440 6177 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.156009 6177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.156150 6177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.156622 6177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.157475 6177 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.157532 6177 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.157586 6177 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.157611 6177 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.165627 6177 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.68:39131
I20240501 19:55:59.165688 6305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.68:39131 every 8 connection(s)
I20240501 19:55:59.166494 6177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:55:59.169684 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 6177
I20240501 19:55:59.169792 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:55:59.171854 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.69:0
--local_ip_for_outbound_sockets=127.5.139.69
--webserver_interface=127.5.139.69
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--builtin_ntp_servers=127.5.139.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:55:59.175114 6306 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35927
I20240501 19:55:59.175308 6306 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.175822 6306 heartbeater.cc:511] Master 127.5.139.126:35927 requested a full tablet report, sending...
I20240501 19:55:59.210363 5719 ts_manager.cc:194] Registered new tserver with Master: fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:55:59.211062 5719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.68:57399
W20240501 19:55:59.273669 6309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.273849 6309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.273914 6309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.276795 6309 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:59.276857 6309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.276947 6309 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.69
I20240501 19:55:59.280375 6309 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.139.69
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--never_fsync=true
--heap_profile_path=/tmp/kudu.6309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.69
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.281144 6309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.282109 6309 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.290120 6315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.290792 6318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.291478 6316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.292209 6309 server_base.cc:1034] running on GCE node
I20240501 19:55:59.292511 6309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.292961 6309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.294111 6309 hybrid_clock.cc:648] HybridClock initialized: now 1714593359294097 us; error 23 us; skew 500 ppm
I20240501 19:55:59.295776 6309 webserver.cc:460] Webserver started at http://127.5.139.69:44073/ using document root <none> and password file <none>
I20240501 19:55:59.296324 6309 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.296380 6309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.296522 6309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.298103 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a2a04e6cd48541839ce83dc1d1790975"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-twc7"
I20240501 19:55:59.298679 6309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a2a04e6cd48541839ce83dc1d1790975"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-twc7"
I20240501 19:55:59.301931 6309 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:59.303974 6324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.304917 6309 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.305020 6309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a2a04e6cd48541839ce83dc1d1790975"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-twc7"
I20240501 19:55:59.305135 6309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.317145 6309 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.317749 6309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.317888 6309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.318377 6309 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.319258 6309 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.319324 6309 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.319361 6309 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.319382 6309 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.328203 6309 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.69:42511
I20240501 19:55:59.328223 6437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.69:42511 every 8 connection(s)
I20240501 19:55:59.329139 6309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:55:59.337000 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 6309
I20240501 19:55:59.337142 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:55:59.337641 6438 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35927
I20240501 19:55:59.337863 6438 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.338371 6438 heartbeater.cc:511] Master 127.5.139.126:35927 requested a full tablet report, sending...
I20240501 19:55:59.339294 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.70:0
--local_ip_for_outbound_sockets=127.5.139.70
--webserver_interface=127.5.139.70
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--builtin_ntp_servers=127.5.139.84:42691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:55:59.372069 5719 ts_manager.cc:194] Registered new tserver with Master: a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:55:59.372793 5719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.69:34037
W20240501 19:55:59.448771 6441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:55:59.448994 6441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:55:59.449066 6441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:55:59.452010 6441 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:55:59.452067 6441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:55:59.452131 6441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.70
I20240501 19:55:59.455658 6441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.139.70
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35927
--never_fsync=true
--heap_profile_path=/tmp/kudu.6441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.70
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:55:59.456502 6441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:55:59.457603 6441 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:55:59.465889 6447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.466584 6450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:55:59.467221 6448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:55:59.467962 6441 server_base.cc:1034] running on GCE node
I20240501 19:55:59.468317 6441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:55:59.468834 6441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:55:59.469986 6441 hybrid_clock.cc:648] HybridClock initialized: now 1714593359469975 us; error 36 us; skew 500 ppm
I20240501 19:55:59.471743 6441 webserver.cc:460] Webserver started at http://127.5.139.70:43881/ using document root <none> and password file <none>
I20240501 19:55:59.472266 6441 fs_manager.cc:359] Metadata directory not provided
I20240501 19:55:59.472322 6441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:55:59.472513 6441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:55:59.474128 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "60b2c1ef2e86405bba90eaa1db142e56"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-twc7"
I20240501 19:55:59.474576 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "60b2c1ef2e86405bba90eaa1db142e56"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-twc7"
I20240501 19:55:59.477804 6441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:55:59.479833 6456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.480763 6441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:55:59.480851 6441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "60b2c1ef2e86405bba90eaa1db142e56"
format_stamp: "Formatted at 2024-05-01 19:55:59 on dist-test-slave-twc7"
I20240501 19:55:59.480916 6441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:55:59.502101 6441 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:55:59.502666 6441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:55:59.502791 6441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:55:59.503242 6441 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:55:59.504045 6441 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:55:59.504089 6441 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.504128 6441 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:55:59.504150 6441 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:55:59.512336 6441 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.70:35011
I20240501 19:55:59.512359 6569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.70:35011 every 8 connection(s)
I20240501 19:55:59.513195 6441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:55:59.515576 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 6441
I20240501 19:55:59.515693 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:55:59.522058 6570 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35927
I20240501 19:55:59.522239 6570 heartbeater.cc:464] Registering TS with master...
I20240501 19:55:59.522682 6570 heartbeater.cc:511] Master 127.5.139.126:35927 requested a full tablet report, sending...
I20240501 19:55:59.555626 5719 ts_manager.cc:194] Registered new tserver with Master: 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011)
I20240501 19:55:59.556260 5719 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.70:46119
I20240501 19:55:59.564913 5677 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 19:55:59.576117 5677 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 19:55:59.679702 5909 heartbeater.cc:503] Master 127.5.139.126:35927 was elected leader, sending a full tablet report...
I20240501 19:55:59.863890 6041 heartbeater.cc:503] Master 127.5.139.126:35927 was elected leader, sending a full tablet report...
I20240501 19:56:00.031966 6173 heartbeater.cc:503] Master 127.5.139.126:35927 was elected leader, sending a full tablet report...
I20240501 19:56:00.213008 6306 heartbeater.cc:503] Master 127.5.139.126:35927 was elected leader, sending a full tablet report...
I20240501 19:56:00.374771 6438 heartbeater.cc:503] Master 127.5.139.126:35927 was elected leader, sending a full tablet report...
I20240501 19:56:00.558498 6570 heartbeater.cc:503] Master 127.5.139.126:35927 was elected leader, sending a full tablet report...
I20240501 19:56:03.328508 5719 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:03.350270 6504 tablet_service.cc:1467] Processing CreateTablet for tablet ed54775aa8764fe18974093dcbb7d49b (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8be85d55d5541d9b21440d53273b979]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.350239 5975 tablet_service.cc:1467] Processing CreateTablet for tablet 48978f6f1980497cb3bd91c26de554d6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8be85d55d5541d9b21440d53273b979]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.350420 6107 tablet_service.cc:1467] Processing CreateTablet for tablet 48978f6f1980497cb3bd91c26de554d6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8be85d55d5541d9b21440d53273b979]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.351199 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48978f6f1980497cb3bd91c26de554d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.351199 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed54775aa8764fe18974093dcbb7d49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.351202 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48978f6f1980497cb3bd91c26de554d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.353440 6106 tablet_service.cc:1467] Processing CreateTablet for tablet 8aede5dd927f4029ad1c9d77dcf073a0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8be85d55d5541d9b21440d53273b979]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.353696 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aede5dd927f4029ad1c9d77dcf073a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.355834 6372 tablet_service.cc:1467] Processing CreateTablet for tablet 48978f6f1980497cb3bd91c26de554d6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8be85d55d5541d9b21440d53273b979]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.355834 6371 tablet_service.cc:1467] Processing CreateTablet for tablet 8aede5dd927f4029ad1c9d77dcf073a0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8be85d55d5541d9b21440d53273b979]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.356640 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48978f6f1980497cb3bd91c26de554d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.357468 6596 tablet_bootstrap.cc:492] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:03.357738 6371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aede5dd927f4029ad1c9d77dcf073a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.358151 6595 tablet_bootstrap.cc:492] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:03.359581 6596 tablet_bootstrap.cc:654] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.360401 6596 log.cc:824] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.361407 6595 tablet_bootstrap.cc:654] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.361446 6240 tablet_service.cc:1467] Processing CreateTablet for tablet ed54775aa8764fe18974093dcbb7d49b (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8be85d55d5541d9b21440d53273b979]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.362035 6596 tablet_bootstrap.cc:492] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:03.362105 6595 log.cc:824] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.362181 6596 ts_tablet_manager.cc:1367] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:56:03.362232 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed54775aa8764fe18974093dcbb7d49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.365866 6596 raft_consensus.cc:348] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.366045 6596 raft_consensus.cc:374] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.366092 6596 raft_consensus.cc:729] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.366541 6596 consensus_queue.cc:260] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.367369 6596 ts_tablet_manager.cc:1398] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20240501 19:56:03.367244 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 8aede5dd927f4029ad1c9d77dcf073a0 (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8be85d55d5541d9b21440d53273b979]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.367902 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8aede5dd927f4029ad1c9d77dcf073a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.368714 6601 tablet_bootstrap.cc:492] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:03.369524 5843 tablet_service.cc:1467] Processing CreateTablet for tablet ed54775aa8764fe18974093dcbb7d49b (DEFAULT_TABLE table=rebalance_test_table_0 [id=d8be85d55d5541d9b21440d53273b979]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.369743 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed54775aa8764fe18974093dcbb7d49b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.370692 6601 tablet_bootstrap.cc:654] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.371419 6601 log.cc:824] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.372541 6603 tablet_bootstrap.cc:492] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:03.374035 6599 tablet_bootstrap.cc:492] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:03.374488 6603 tablet_bootstrap.cc:654] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.375228 6603 log.cc:824] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.375334 6604 tablet_bootstrap.cc:492] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:03.375769 6599 tablet_bootstrap.cc:654] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.376319 6599 log.cc:824] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.376873 6604 tablet_bootstrap.cc:654] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.377586 6604 log.cc:824] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:03.377825 6601 tablet_bootstrap.cc:492] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:03.377885 6595 tablet_bootstrap.cc:492] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:03.377983 6601 ts_tablet_manager.cc:1367] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 19:56:03.378013 6595 ts_tablet_manager.cc:1367] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20240501 19:56:03.378017 6599 tablet_bootstrap.cc:492] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:03.378165 6599 ts_tablet_manager.cc:1367] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:03.379212 6604 tablet_bootstrap.cc:492] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:03.379397 6604 ts_tablet_manager.cc:1367] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:03.381592 6595 raft_consensus.cc:348] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.381592 6601 raft_consensus.cc:348] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.381795 6601 raft_consensus.cc:374] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.381789 6595 raft_consensus.cc:374] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.381845 6595 raft_consensus.cc:729] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.382256 6595 consensus_queue.cc:260] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.382519 6599 raft_consensus.cc:348] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.382614 6601 raft_consensus.cc:729] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.382741 6599 raft_consensus.cc:374] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.382822 6599 raft_consensus.cc:729] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.383133 6595 ts_tablet_manager.cc:1398] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:03.383178 6601 consensus_queue.cc:260] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.383307 6599 consensus_queue.cc:260] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.383492 6595 tablet_bootstrap.cc:492] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:03.384202 6601 ts_tablet_manager.cc:1398] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240501 19:56:03.384251 6599 ts_tablet_manager.cc:1398] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:56:03.384433 6595 tablet_bootstrap.cc:654] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.384824 6599 tablet_bootstrap.cc:492] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:03.385725 6599 tablet_bootstrap.cc:654] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.387285 6595 tablet_bootstrap.cc:492] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:03.386878 6604 raft_consensus.cc:348] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.387348 6604 raft_consensus.cc:374] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.387375 6595 ts_tablet_manager.cc:1367] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:03.387396 6604 raft_consensus.cc:729] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.387522 6603 tablet_bootstrap.cc:492] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:03.387658 6603 ts_tablet_manager.cc:1367] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 19:56:03.387786 6604 consensus_queue.cc:260] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.387863 6595 raft_consensus.cc:348] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.387956 6595 raft_consensus.cc:374] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.387984 6595 raft_consensus.cc:729] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.388104 6595 consensus_queue.cc:260] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.388337 6595 ts_tablet_manager.cc:1398] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.388432 6604 ts_tablet_manager.cc:1398] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20240501 19:56:03.388633 6610 raft_consensus.cc:482] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.388749 6610 raft_consensus.cc:504] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.389801 6610 leader_election.cc:288] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845)
I20240501 19:56:03.391360 6603 raft_consensus.cc:348] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.391574 6603 raft_consensus.cc:374] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.391669 6603 raft_consensus.cc:729] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.392016 6603 consensus_queue.cc:260] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.393119 6603 ts_tablet_manager.cc:1398] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:03.393446 6603 tablet_bootstrap.cc:492] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:03.393888 6599 tablet_bootstrap.cc:492] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:03.393972 6599 ts_tablet_manager.cc:1367] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:56:03.394516 6599 raft_consensus.cc:348] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.394620 6599 raft_consensus.cc:374] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.394652 6599 raft_consensus.cc:729] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.394753 6599 consensus_queue.cc:260] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.395072 6599 ts_tablet_manager.cc:1398] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.395119 6603 tablet_bootstrap.cc:654] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.396941 6603 tablet_bootstrap.cc:492] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:03.397399 6603 ts_tablet_manager.cc:1367] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:03.397691 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aede5dd927f4029ad1c9d77dcf073a0" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:03.397675 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aede5dd927f4029ad1c9d77dcf073a0" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:03.397889 6127 raft_consensus.cc:2454] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:03.397970 6603 raft_consensus.cc:348] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.398036 6603 raft_consensus.cc:374] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.398064 6603 raft_consensus.cc:729] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.398226 6603 consensus_queue.cc:260] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
W20240501 19:56:03.398411 6328 leader_election.cc:341] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845): Illegal state: must be running to vote when last-logged opid is not known
I20240501 19:56:03.398458 6603 ts_tablet_manager.cc:1398] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.398698 6327 leader_election.cc:302] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters: 4629e756401746a8bf4652f995bddb44
I20240501 19:56:03.399081 6613 raft_consensus.cc:2781] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.399165 6613 raft_consensus.cc:482] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.399262 6613 raft_consensus.cc:3037] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.400357 6613 raft_consensus.cc:504] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.400969 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aede5dd927f4029ad1c9d77dcf073a0" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44"
I20240501 19:56:03.400969 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8aede5dd927f4029ad1c9d77dcf073a0" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:03.401086 6127 raft_consensus.cc:3037] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.401103 5863 raft_consensus.cc:3037] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.402079 6127 raft_consensus.cc:2454] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:03.402083 5863 raft_consensus.cc:2454] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:03.402324 6613 leader_election.cc:288] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Requested vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845)
I20240501 19:56:03.402343 6328 leader_election.cc:302] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:03.402479 6613 raft_consensus.cc:2781] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.402561 6613 raft_consensus.cc:686] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [term 1 LEADER]: Becoming Leader. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Running, Role: LEADER
I20240501 19:56:03.402913 6613 consensus_queue.cc:237] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.405367 5720 catalog_manager.cc:5484] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69). New cstate: current_term: 1 leader_uuid: "a2a04e6cd48541839ce83dc1d1790975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:03.420002 6307 tablet.cc:2369] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:03.477555 6617 raft_consensus.cc:482] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.477694 6617 raft_consensus.cc:504] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.478737 6617 leader_election.cc:288] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:03.482400 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed54775aa8764fe18974093dcbb7d49b" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:03.482407 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed54775aa8764fe18974093dcbb7d49b" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:03.482666 6524 raft_consensus.cc:2454] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:03.482667 6260 raft_consensus.cc:2454] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:03.483397 5798 leader_election.cc:302] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:03.483621 6617 raft_consensus.cc:2781] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.483685 6617 raft_consensus.cc:482] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.483770 6617 raft_consensus.cc:3037] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.484620 6617 raft_consensus.cc:504] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:03.484934 6617 leader_election.cc:288] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:03.485342 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed54775aa8764fe18974093dcbb7d49b" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:03.485354 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed54775aa8764fe18974093dcbb7d49b" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:03.485441 6260 raft_consensus.cc:3037] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.485445 6524 raft_consensus.cc:3037] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.486234 6260 raft_consensus.cc:2454] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:03.486299 6524 raft_consensus.cc:2454] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:03.486539 5799 leader_election.cc:302] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:03.486755 6617 raft_consensus.cc:2781] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.486941 6617 raft_consensus.cc:686] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [term 1 LEADER]: Becoming Leader. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Running, Role: LEADER
I20240501 19:56:03.487218 6617 consensus_queue.cc:237] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
W20240501 19:56:03.487962 6174 tablet.cc:2369] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:03.489933 5720 catalog_manager.cc:5484] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4629e756401746a8bf4652f995bddb44 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "4629e756401746a8bf4652f995bddb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:03.516600 6571 tablet.cc:2369] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:03.573184 6042 tablet.cc:2369] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:03.577840 6609 raft_consensus.cc:482] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.578022 6609 raft_consensus.cc:504] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.579270 6609 leader_election.cc:288] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
W20240501 19:56:03.582731 6439 tablet.cc:2369] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:03.584118 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48978f6f1980497cb3bd91c26de554d6" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:03.584268 6127 raft_consensus.cc:2454] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:03.584728 5930 leader_election.cc:302] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:03.585207 6609 raft_consensus.cc:2781] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.585304 6609 raft_consensus.cc:482] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.585359 6609 raft_consensus.cc:3037] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.585655 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48978f6f1980497cb3bd91c26de554d6" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:03.585845 6392 raft_consensus.cc:2454] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:03.586642 6609 raft_consensus.cc:504] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.587064 6609 leader_election.cc:288] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Requested vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:03.587311 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48978f6f1980497cb3bd91c26de554d6" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:03.587445 6127 raft_consensus.cc:3037] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.588382 6127 raft_consensus.cc:2454] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:03.588774 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48978f6f1980497cb3bd91c26de554d6" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:03.588830 5930 leader_election.cc:302] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:03.588888 6392 raft_consensus.cc:3037] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.589130 6609 raft_consensus.cc:2781] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.589355 6609 raft_consensus.cc:686] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [term 1 LEADER]: Becoming Leader. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Running, Role: LEADER
I20240501 19:56:03.589680 6609 consensus_queue.cc:237] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.589776 6392 raft_consensus.cc:2454] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:03.592872 5720 catalog_manager.cc:5484] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3489b69a0428491e957e46effb9b1146 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "3489b69a0428491e957e46effb9b1146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:03.619094 5910 tablet.cc:2369] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:03.748040 6628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:03.748246 6628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:03.751173 6628 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 19:56:03.841421 6392 raft_consensus.cc:1261] T 48978f6f1980497cb3bd91c26de554d6 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.842259 6127 raft_consensus.cc:1261] T 48978f6f1980497cb3bd91c26de554d6 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.842379 6626 consensus_queue.cc:1035] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.843118 6626 consensus_queue.cc:1035] T 48978f6f1980497cb3bd91c26de554d6 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.844620 6260 raft_consensus.cc:1261] T ed54775aa8764fe18974093dcbb7d49b P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.845341 6524 raft_consensus.cc:1261] T ed54775aa8764fe18974093dcbb7d49b P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.846128 6622 consensus_queue.cc:1035] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.846551 6622 consensus_queue.cc:1035] T ed54775aa8764fe18974093dcbb7d49b P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.850952 6655 mvcc.cc:205] Tried to move back new op lower bound from 7022974418285113344 to 7022974417266409472. Current Snapshot: MvccSnapshot[applied={T|T < 7022974418285113344}]
I20240501 19:56:03.851687 5863 raft_consensus.cc:1261] T 8aede5dd927f4029ad1c9d77dcf073a0 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.852200 6126 raft_consensus.cc:1261] T 8aede5dd927f4029ad1c9d77dcf073a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:03.852248 6613 consensus_queue.cc:1035] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.852874 6660 mvcc.cc:205] Tried to move back new op lower bound from 7022974418285113344 to 7022974417266409472. Current Snapshot: MvccSnapshot[applied={T|T < 7022974418285113344}]
I20240501 19:56:03.853850 6613 consensus_queue.cc:1035] T 8aede5dd927f4029ad1c9d77dcf073a0 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:03.867007 6670 mvcc.cc:205] Tried to move back new op lower bound from 7022974418323456000 to 7022974416500555776. Current Snapshot: MvccSnapshot[applied={T|T < 7022974418323456000}]
I20240501 19:56:03.867260 6656 mvcc.cc:205] Tried to move back new op lower bound from 7022974418315685888 to 7022974416845864960. Current Snapshot: MvccSnapshot[applied={T|T < 7022974418296717312}]
I20240501 19:56:03.875465 6664 mvcc.cc:205] Tried to move back new op lower bound from 7022974418296717312 to 7022974416845864960. Current Snapshot: MvccSnapshot[applied={T|T < 7022974418296717312}]
I20240501 19:56:03.899057 6676 mvcc.cc:205] Tried to move back new op lower bound from 7022974418296717312 to 7022974416845864960. Current Snapshot: MvccSnapshot[applied={T|T < 7022974418296717312}]
INSERT report
rows total: 2000
time total: 126.075 ms
time per row: 0.0630376 ms
I20240501 19:56:03.920665 5718 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:03.925932 6240 tablet_service.cc:1467] Processing CreateTablet for tablet a6161b2d95084e9c90fb83929d2d1c70 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00e00d9a3594d239c53a9e657345e48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.925981 6504 tablet_service.cc:1467] Processing CreateTablet for tablet a6161b2d95084e9c90fb83929d2d1c70 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00e00d9a3594d239c53a9e657345e48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.926254 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6161b2d95084e9c90fb83929d2d1c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.926254 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6161b2d95084e9c90fb83929d2d1c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.926961 6107 tablet_service.cc:1467] Processing CreateTablet for tablet 980bc7bedd7d48d083b2c0b951339289 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00e00d9a3594d239c53a9e657345e48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.927075 6372 tablet_service.cc:1467] Processing CreateTablet for tablet 980bc7bedd7d48d083b2c0b951339289 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00e00d9a3594d239c53a9e657345e48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.927232 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 980bc7bedd7d48d083b2c0b951339289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.927335 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 980bc7bedd7d48d083b2c0b951339289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.928486 6692 tablet_bootstrap.cc:492] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:03.928519 5975 tablet_service.cc:1467] Processing CreateTablet for tablet a6161b2d95084e9c90fb83929d2d1c70 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00e00d9a3594d239c53a9e657345e48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.928754 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6161b2d95084e9c90fb83929d2d1c70. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.929322 6692 tablet_bootstrap.cc:654] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.929643 6693 tablet_bootstrap.cc:492] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:03.930483 6693 tablet_bootstrap.cc:654] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.930513 6692 tablet_bootstrap.cc:492] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:03.930588 6692 ts_tablet_manager.cc:1367] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:03.930887 6696 tablet_bootstrap.cc:492] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:03.931056 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 607e3347b7934e08a169f1a187941f6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00e00d9a3594d239c53a9e657345e48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.931120 6692 raft_consensus.cc:348] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.931221 6692 raft_consensus.cc:374] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.931258 6692 raft_consensus.cc:729] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.931313 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607e3347b7934e08a169f1a187941f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.931329 6694 tablet_bootstrap.cc:492] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:03.931387 6692 consensus_queue.cc:260] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.931774 6692 ts_tablet_manager.cc:1398] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.932111 6696 tablet_bootstrap.cc:654] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.932600 6694 tablet_bootstrap.cc:654] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.933318 6696 tablet_bootstrap.cc:492] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:03.933406 6696 ts_tablet_manager.cc:1367] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:03.933627 6240 tablet_service.cc:1467] Processing CreateTablet for tablet 607e3347b7934e08a169f1a187941f6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00e00d9a3594d239c53a9e657345e48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.933759 6693 tablet_bootstrap.cc:492] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:03.933830 6693 ts_tablet_manager.cc:1367] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:03.933848 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607e3347b7934e08a169f1a187941f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.933809 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 980bc7bedd7d48d083b2c0b951339289 (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00e00d9a3594d239c53a9e657345e48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.933938 6696 raft_consensus.cc:348] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.934022 6696 raft_consensus.cc:374] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.934051 6696 raft_consensus.cc:729] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.934051 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 980bc7bedd7d48d083b2c0b951339289. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.934181 6696 consensus_queue.cc:260] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.934428 6696 ts_tablet_manager.cc:1398] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.934242 6693 raft_consensus.cc:348] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.934541 6693 raft_consensus.cc:374] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.934576 6693 raft_consensus.cc:729] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.934845 6696 tablet_bootstrap.cc:492] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:03.934691 6693 consensus_queue.cc:260] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.935197 6693 ts_tablet_manager.cc:1398] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.936229 6700 tablet_bootstrap.cc:492] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:03.937222 6700 tablet_bootstrap.cc:654] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.937400 6696 tablet_bootstrap.cc:654] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.938338 6694 tablet_bootstrap.cc:492] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:03.938423 6694 ts_tablet_manager.cc:1367] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:03.938477 6504 tablet_service.cc:1467] Processing CreateTablet for tablet 607e3347b7934e08a169f1a187941f6b (DEFAULT_TABLE table=rebalance_test_table_1 [id=d00e00d9a3594d239c53a9e657345e48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:03.938686 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 607e3347b7934e08a169f1a187941f6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:03.938745 6696 tablet_bootstrap.cc:492] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:03.938834 6696 ts_tablet_manager.cc:1367] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:03.938844 6694 raft_consensus.cc:348] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.938946 6694 raft_consensus.cc:374] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.938977 6694 raft_consensus.cc:729] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.939081 6694 consensus_queue.cc:260] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.939217 6696 raft_consensus.cc:348] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.939294 6696 raft_consensus.cc:374] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.939322 6696 raft_consensus.cc:729] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.939349 6694 ts_tablet_manager.cc:1398] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.939422 6696 consensus_queue.cc:260] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.939637 6696 ts_tablet_manager.cc:1398] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:03.941321 6692 tablet_bootstrap.cc:492] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:03.942186 6692 tablet_bootstrap.cc:654] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.942574 6700 tablet_bootstrap.cc:492] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:03.942656 6700 ts_tablet_manager.cc:1367] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:03.943100 6700 raft_consensus.cc:348] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.943183 6700 raft_consensus.cc:374] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.943205 6700 raft_consensus.cc:729] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.943245 6692 tablet_bootstrap.cc:492] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:03.943331 6692 ts_tablet_manager.cc:1367] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:03.943306 6700 consensus_queue.cc:260] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.943671 6692 raft_consensus.cc:348] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.943775 6692 raft_consensus.cc:374] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.943822 6692 raft_consensus.cc:729] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.943851 6700 ts_tablet_manager.cc:1398] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.943964 6692 consensus_queue.cc:260] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.944195 6692 ts_tablet_manager.cc:1398] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:03.944939 6691 tablet_bootstrap.cc:492] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:03.945706 6691 tablet_bootstrap.cc:654] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.946667 6691 tablet_bootstrap.cc:492] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:03.946740 6691 ts_tablet_manager.cc:1367] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:03.947157 6691 raft_consensus.cc:348] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.947243 6691 raft_consensus.cc:374] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.947275 6691 raft_consensus.cc:729] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.947427 6691 consensus_queue.cc:260] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.947659 6691 ts_tablet_manager.cc:1398] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:03.947842 6691 tablet_bootstrap.cc:492] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:03.948832 6691 tablet_bootstrap.cc:654] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:03.949978 6691 tablet_bootstrap.cc:492] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:03.950088 6691 ts_tablet_manager.cc:1367] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:03.950484 6691 raft_consensus.cc:348] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.950567 6691 raft_consensus.cc:374] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:03.950588 6691 raft_consensus.cc:729] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:03.950701 6691 consensus_queue.cc:260] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.950947 6691 ts_tablet_manager.cc:1398] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:03.986964 6663 raft_consensus.cc:482] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.987108 6663 raft_consensus.cc:504] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.987519 6663 leader_election.cc:288] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:03.987848 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980bc7bedd7d48d083b2c0b951339289" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:03.987879 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980bc7bedd7d48d083b2c0b951339289" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:03.987984 6127 raft_consensus.cc:2454] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:03.987993 5863 raft_consensus.cc:2454] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:03.988281 6327 leader_election.cc:302] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:03.988515 6663 raft_consensus.cc:2781] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.988574 6663 raft_consensus.cc:482] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.988622 6663 raft_consensus.cc:3037] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.989645 6663 raft_consensus.cc:504] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.989980 6663 leader_election.cc:288] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Requested vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:03.990258 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980bc7bedd7d48d083b2c0b951339289" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:03.990298 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "980bc7bedd7d48d083b2c0b951339289" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44"
I20240501 19:56:03.990358 6127 raft_consensus.cc:3037] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.990402 5863 raft_consensus.cc:3037] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.991070 6127 raft_consensus.cc:2454] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:03.991232 5863 raft_consensus.cc:2454] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:03.991369 6327 leader_election.cc:302] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:03.991557 6663 raft_consensus.cc:2781] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:03.991639 6663 raft_consensus.cc:686] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [term 1 LEADER]: Becoming Leader. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Running, Role: LEADER
I20240501 19:56:03.991788 6663 consensus_queue.cc:237] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:03.992085 6614 raft_consensus.cc:482] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:03.992188 6614 raft_consensus.cc:504] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.993302 6614 leader_election.cc:288] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:03.993492 5717 catalog_manager.cc:5484] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69). New cstate: current_term: 1 leader_uuid: "a2a04e6cd48541839ce83dc1d1790975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:03.998076 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607e3347b7934e08a169f1a187941f6b" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:03.998209 6524 raft_consensus.cc:2454] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:03.998504 6195 leader_election.cc:302] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:03.998740 6614 raft_consensus.cc:2781] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:03.998800 6614 raft_consensus.cc:482] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:03.998850 6614 raft_consensus.cc:3037] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:03.999572 6614 raft_consensus.cc:504] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:03.999868 6614 leader_election.cc:288] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:04.000922 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607e3347b7934e08a169f1a187941f6b" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:04.001057 6524 raft_consensus.cc:3037] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.001106 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607e3347b7934e08a169f1a187941f6b" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:04.001116 5995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "607e3347b7934e08a169f1a187941f6b" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:04.001312 5994 raft_consensus.cc:3037] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.002002 6524 raft_consensus.cc:2454] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:04.002205 5994 raft_consensus.cc:2454] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:04.002272 6195 leader_election.cc:302] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:04.002453 6614 raft_consensus.cc:2781] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.002651 6614 raft_consensus.cc:686] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [term 1 LEADER]: Becoming Leader. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Running, Role: LEADER
I20240501 19:56:04.002895 6614 consensus_queue.cc:237] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.004745 5717 catalog_manager.cc:5484] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1af421bf6a480587c69697af9f39dd (127.5.139.68). New cstate: current_term: 1 leader_uuid: "fb1af421bf6a480587c69697af9f39dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:04.052290 6614 raft_consensus.cc:482] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.052429 6614 raft_consensus.cc:504] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.052793 6614 leader_election.cc:288] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:04.053320 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6161b2d95084e9c90fb83929d2d1c70" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:04.053354 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6161b2d95084e9c90fb83929d2d1c70" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:04.053476 6524 raft_consensus.cc:2454] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:04.053491 5994 raft_consensus.cc:2454] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:04.053781 6195 leader_election.cc:302] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:04.053997 6614 raft_consensus.cc:2781] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.054065 6614 raft_consensus.cc:482] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.054122 6614 raft_consensus.cc:3037] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.054942 6614 raft_consensus.cc:504] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.055269 6614 leader_election.cc:288] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:04.055652 6524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6161b2d95084e9c90fb83929d2d1c70" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:04.055686 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6161b2d95084e9c90fb83929d2d1c70" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:04.055781 6524 raft_consensus.cc:3037] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.055799 5994 raft_consensus.cc:3037] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.056700 5994 raft_consensus.cc:2454] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:04.056699 6524 raft_consensus.cc:2454] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:04.056996 6195 leader_election.cc:302] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:04.057171 6614 raft_consensus.cc:2781] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.057256 6614 raft_consensus.cc:686] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [term 1 LEADER]: Becoming Leader. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Running, Role: LEADER
I20240501 19:56:04.057400 6614 consensus_queue.cc:237] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.059087 5717 catalog_manager.cc:5484] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1af421bf6a480587c69697af9f39dd (127.5.139.68). New cstate: current_term: 1 leader_uuid: "fb1af421bf6a480587c69697af9f39dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:04.197959 6713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.198197 6713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.201191 6713 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 19:56:04.287142 5863 raft_consensus.cc:1261] T 980bc7bedd7d48d083b2c0b951339289 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.287328 6127 raft_consensus.cc:1261] T 980bc7bedd7d48d083b2c0b951339289 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.287742 6610 consensus_queue.cc:1035] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.288080 6524 raft_consensus.cc:1261] T 607e3347b7934e08a169f1a187941f6b P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.289441 6610 consensus_queue.cc:1035] T 980bc7bedd7d48d083b2c0b951339289 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.290345 5994 raft_consensus.cc:1261] T 607e3347b7934e08a169f1a187941f6b P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.290910 6710 consensus_queue.cc:1035] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.291298 6710 consensus_queue.cc:1035] T 607e3347b7934e08a169f1a187941f6b P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.293406 5995 raft_consensus.cc:1261] T a6161b2d95084e9c90fb83929d2d1c70 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.294332 6523 raft_consensus.cc:1261] T a6161b2d95084e9c90fb83929d2d1c70 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.294610 6614 consensus_queue.cc:1035] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.295073 6614 consensus_queue.cc:1035] T a6161b2d95084e9c90fb83929d2d1c70 P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.64 ms
time per row: 0.0568199 ms
I20240501 19:56:04.360196 5715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:04.365945 6372 tablet_service.cc:1467] Processing CreateTablet for tablet d30c0fccb70845978605837176173e6f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee400b237e8f48668ac027f51b59ac81]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.366309 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30c0fccb70845978605837176173e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.366570 6240 tablet_service.cc:1467] Processing CreateTablet for tablet cec70d5cb2c64222ad47ecc922be3da6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee400b237e8f48668ac027f51b59ac81]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.366691 5842 tablet_service.cc:1467] Processing CreateTablet for tablet d30c0fccb70845978605837176173e6f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee400b237e8f48668ac027f51b59ac81]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.366879 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec70d5cb2c64222ad47ecc922be3da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.366930 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30c0fccb70845978605837176173e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.367416 6107 tablet_service.cc:1467] Processing CreateTablet for tablet d30c0fccb70845978605837176173e6f (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee400b237e8f48668ac027f51b59ac81]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.367710 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d30c0fccb70845978605837176173e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.368743 6694 tablet_bootstrap.cc:492] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:04.369122 6691 tablet_bootstrap.cc:492] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:04.369086 5974 tablet_service.cc:1467] Processing CreateTablet for tablet cec70d5cb2c64222ad47ecc922be3da6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee400b237e8f48668ac027f51b59ac81]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.369347 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec70d5cb2c64222ad47ecc922be3da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.369657 6694 tablet_bootstrap.cc:654] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.369926 6691 tablet_bootstrap.cc:654] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.369944 6693 tablet_bootstrap.cc:492] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:04.370633 6700 tablet_bootstrap.cc:492] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:04.370807 6693 tablet_bootstrap.cc:654] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.371400 6696 tablet_bootstrap.cc:492] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:04.371521 6700 tablet_bootstrap.cc:654] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.371977 6693 tablet_bootstrap.cc:492] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:04.372053 6693 ts_tablet_manager.cc:1367] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.372254 6696 tablet_bootstrap.cc:654] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.372326 6504 tablet_service.cc:1467] Processing CreateTablet for tablet cec70d5cb2c64222ad47ecc922be3da6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee400b237e8f48668ac027f51b59ac81]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.372604 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cec70d5cb2c64222ad47ecc922be3da6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.372517 6693 raft_consensus.cc:348] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:04.372642 6693 raft_consensus.cc:374] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.372680 6693 raft_consensus.cc:729] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.372813 6693 consensus_queue.cc:260] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:04.373093 6693 ts_tablet_manager.cc:1398] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.373077 5843 tablet_service.cc:1467] Processing CreateTablet for tablet c1e625cabd1845bda82e7b78317fe081 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee400b237e8f48668ac027f51b59ac81]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.373284 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e625cabd1845bda82e7b78317fe081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.373303 6691 tablet_bootstrap.cc:492] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:04.373375 6691 ts_tablet_manager.cc:1367] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:04.373812 6691 raft_consensus.cc:348] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:04.373909 6691 raft_consensus.cc:374] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.373941 6691 raft_consensus.cc:729] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.374048 6691 consensus_queue.cc:260] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:04.374306 6691 ts_tablet_manager.cc:1398] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.374831 6692 tablet_bootstrap.cc:492] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:04.375677 6692 tablet_bootstrap.cc:654] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.375895 6696 tablet_bootstrap.cc:492] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:04.375969 6696 ts_tablet_manager.cc:1367] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:04.375910 6107 tablet_service.cc:1467] Processing CreateTablet for tablet c1e625cabd1845bda82e7b78317fe081 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee400b237e8f48668ac027f51b59ac81]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.376132 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e625cabd1845bda82e7b78317fe081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.376384 6696 raft_consensus.cc:348] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:04.376480 6696 raft_consensus.cc:374] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.376520 6696 raft_consensus.cc:729] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.376638 6696 consensus_queue.cc:260] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:04.376886 6696 ts_tablet_manager.cc:1398] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.377243 6694 tablet_bootstrap.cc:492] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:04.377344 6694 ts_tablet_manager.cc:1367] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:04.377790 6694 raft_consensus.cc:348] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:04.377893 6694 raft_consensus.cc:374] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.377923 6694 raft_consensus.cc:729] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.378048 6694 consensus_queue.cc:260] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:04.378315 6694 ts_tablet_manager.cc:1398] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.378846 6693 tablet_bootstrap.cc:492] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:04.378891 6692 tablet_bootstrap.cc:492] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:04.378968 6692 ts_tablet_manager.cc:1367] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:04.379328 6700 tablet_bootstrap.cc:492] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:04.379403 6700 ts_tablet_manager.cc:1367] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:04.379582 6692 raft_consensus.cc:348] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:04.379671 6692 raft_consensus.cc:374] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.379702 6692 raft_consensus.cc:729] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.379755 6693 tablet_bootstrap.cc:654] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.379814 6692 consensus_queue.cc:260] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:04.379843 6700 raft_consensus.cc:348] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:04.379946 6700 raft_consensus.cc:374] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.379979 6700 raft_consensus.cc:729] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.380074 6692 ts_tablet_manager.cc:1398] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.380105 6700 consensus_queue.cc:260] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:04.380360 6700 ts_tablet_manager.cc:1398] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.380993 6700 tablet_bootstrap.cc:492] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:04.381136 6372 tablet_service.cc:1467] Processing CreateTablet for tablet c1e625cabd1845bda82e7b78317fe081 (DEFAULT_TABLE table=rebalance_test_table_2 [id=ee400b237e8f48668ac027f51b59ac81]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.381373 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1e625cabd1845bda82e7b78317fe081. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.382234 6700 tablet_bootstrap.cc:654] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.382365 6693 tablet_bootstrap.cc:492] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:04.382452 6693 ts_tablet_manager.cc:1367] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:04.382684 6610 raft_consensus.cc:482] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.382767 6610 raft_consensus.cc:504] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:04.382833 6693 raft_consensus.cc:348] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.382923 6693 raft_consensus.cc:374] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.382961 6693 raft_consensus.cc:729] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.383080 6693 consensus_queue.cc:260] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.383145 6610 leader_election.cc:288] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:04.383358 6700 tablet_bootstrap.cc:492] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:04.383356 6693 ts_tablet_manager.cc:1398] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.383455 6700 ts_tablet_manager.cc:1367] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:04.383525 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30c0fccb70845978605837176173e6f" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:04.383627 5863 raft_consensus.cc:2454] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:04.383903 6700 raft_consensus.cc:348] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.383953 6328 leader_election.cc:302] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:04.383994 6700 raft_consensus.cc:374] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.383981 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30c0fccb70845978605837176173e6f" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:04.384034 6700 raft_consensus.cc:729] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.384085 6127 raft_consensus.cc:2454] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:04.384105 6610 raft_consensus.cc:2781] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.384145 6610 raft_consensus.cc:482] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.384181 6610 raft_consensus.cc:3037] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.384164 6700 consensus_queue.cc:260] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.384426 6700 ts_tablet_manager.cc:1398] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.385177 6610 raft_consensus.cc:504] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:04.385517 6610 leader_election.cc:288] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Requested vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:04.385927 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30c0fccb70845978605837176173e6f" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44"
I20240501 19:56:04.386018 5863 raft_consensus.cc:3037] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.386870 5863 raft_consensus.cc:2454] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:04.387113 6694 tablet_bootstrap.cc:492] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:04.387151 6328 leader_election.cc:302] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:04.387301 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d30c0fccb70845978605837176173e6f" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:04.387327 6610 raft_consensus.cc:2781] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.387405 6127 raft_consensus.cc:3037] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.387415 6610 raft_consensus.cc:686] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [term 1 LEADER]: Becoming Leader. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Running, Role: LEADER
I20240501 19:56:04.387533 6610 consensus_queue.cc:237] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:04.388191 6127 raft_consensus.cc:2454] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:04.388365 6694 tablet_bootstrap.cc:654] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.389247 5715 catalog_manager.cc:5484] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69). New cstate: current_term: 1 leader_uuid: "a2a04e6cd48541839ce83dc1d1790975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:04.389529 6694 tablet_bootstrap.cc:492] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:04.389662 6694 ts_tablet_manager.cc:1367] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:04.390067 6694 raft_consensus.cc:348] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.390148 6694 raft_consensus.cc:374] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.390170 6694 raft_consensus.cc:729] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.390283 6694 consensus_queue.cc:260] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.390552 6694 ts_tablet_manager.cc:1398] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:04.436789 6613 raft_consensus.cc:482] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.436928 6613 raft_consensus.cc:504] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.437355 6613 leader_election.cc:288] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:04.437749 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e625cabd1845bda82e7b78317fe081" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:04.437769 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e625cabd1845bda82e7b78317fe081" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:04.437892 6127 raft_consensus.cc:2454] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:04.437925 5863 raft_consensus.cc:2454] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:04.438246 6327 leader_election.cc:302] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:04.438416 6613 raft_consensus.cc:2781] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.438478 6613 raft_consensus.cc:482] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.438525 6613 raft_consensus.cc:3037] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.439335 6613 raft_consensus.cc:504] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.439637 6613 leader_election.cc:288] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Requested vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:04.439940 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e625cabd1845bda82e7b78317fe081" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44"
I20240501 19:56:04.439954 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1e625cabd1845bda82e7b78317fe081" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:04.440045 6127 raft_consensus.cc:3037] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.440070 5863 raft_consensus.cc:3037] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.440758 6127 raft_consensus.cc:2454] T c1e625cabd1845bda82e7b78317fe081 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:04.440816 5863 raft_consensus.cc:2454] T c1e625cabd1845bda82e7b78317fe081 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:04.441102 6327 leader_election.cc:302] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:04.441283 6613 raft_consensus.cc:2781] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.441371 6613 raft_consensus.cc:686] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [term 1 LEADER]: Becoming Leader. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Running, Role: LEADER
I20240501 19:56:04.441498 6613 consensus_queue.cc:237] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.443045 5715 catalog_manager.cc:5484] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69). New cstate: current_term: 1 leader_uuid: "a2a04e6cd48541839ce83dc1d1790975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:04.608502 6609 raft_consensus.cc:482] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.608645 6609 raft_consensus.cc:504] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:04.609208 6609 leader_election.cc:288] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:04.612901 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec70d5cb2c64222ad47ecc922be3da6" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:04.613073 6260 raft_consensus.cc:2454] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:04.613242 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec70d5cb2c64222ad47ecc922be3da6" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:04.613401 6523 raft_consensus.cc:2454] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:04.613392 5931 leader_election.cc:302] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:04.613566 6609 raft_consensus.cc:2781] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.613627 6609 raft_consensus.cc:482] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.613665 6609 raft_consensus.cc:3037] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.614483 6609 raft_consensus.cc:504] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:04.614777 6609 leader_election.cc:288] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:04.615111 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec70d5cb2c64222ad47ecc922be3da6" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:04.615128 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cec70d5cb2c64222ad47ecc922be3da6" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:04.615231 6523 raft_consensus.cc:3037] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.615268 6260 raft_consensus.cc:3037] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.616016 6260 raft_consensus.cc:2454] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:04.616132 6523 raft_consensus.cc:2454] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:04.616248 5931 leader_election.cc:302] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:04.616432 6609 raft_consensus.cc:2781] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.616506 6609 raft_consensus.cc:686] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [term 1 LEADER]: Becoming Leader. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Running, Role: LEADER
I20240501 19:56:04.616670 6609 consensus_queue.cc:237] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:04.618378 5715 catalog_manager.cc:5484] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3489b69a0428491e957e46effb9b1146 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "3489b69a0428491e957e46effb9b1146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:04.774593 6771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:04.774822 6771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:04.777791 6771 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 19:56:04.853233 6613 consensus_queue.cc:1035] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.859608 6613 consensus_queue.cc:1035] T c1e625cabd1845bda82e7b78317fe081 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.865839 6260 raft_consensus.cc:1261] T cec70d5cb2c64222ad47ecc922be3da6 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.866300 6609 consensus_queue.cc:1035] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.866721 6523 raft_consensus.cc:1261] T cec70d5cb2c64222ad47ecc922be3da6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.867179 6609 consensus_queue.cc:1035] T cec70d5cb2c64222ad47ecc922be3da6 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.869498 5863 raft_consensus.cc:1261] T d30c0fccb70845978605837176173e6f P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.869855 6127 raft_consensus.cc:1261] T d30c0fccb70845978605837176173e6f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:04.870019 6610 consensus_queue.cc:1035] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:04.870325 6610 consensus_queue.cc:1035] T d30c0fccb70845978605837176173e6f P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 113.447 ms
time per row: 0.0567234 ms
I20240501 19:56:04.931942 5716 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:04.937031 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc7c03eb2f447eb9d268b4cc7036dc3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9bf893cef55841b394f673321b65c634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.937197 6504 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc7c03eb2f447eb9d268b4cc7036dc3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9bf893cef55841b394f673321b65c634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.937364 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc7c03eb2f447eb9d268b4cc7036dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.937435 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc7c03eb2f447eb9d268b4cc7036dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.937470 6240 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc7c03eb2f447eb9d268b4cc7036dc3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9bf893cef55841b394f673321b65c634]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.937754 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc7c03eb2f447eb9d268b4cc7036dc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.939337 6836 tablet_bootstrap.cc:492] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:04.939579 6239 tablet_service.cc:1467] Processing CreateTablet for tablet c37bf45daf71453c816e5247765220e4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9bf893cef55841b394f673321b65c634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.939805 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37bf45daf71453c816e5247765220e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.940166 5843 tablet_service.cc:1467] Processing CreateTablet for tablet e17e4c0e86564147908d01602f82480a (DEFAULT_TABLE table=rebalance_test_table_3 [id=9bf893cef55841b394f673321b65c634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.940268 6836 tablet_bootstrap.cc:654] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.940465 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e4c0e86564147908d01602f82480a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.941267 6836 tablet_bootstrap.cc:492] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:04.941370 6836 ts_tablet_manager.cc:1367] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.941821 6836 raft_consensus.cc:348] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:04.941928 6836 raft_consensus.cc:374] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.941958 6836 raft_consensus.cc:729] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.942082 6836 consensus_queue.cc:260] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:04.942672 6840 tablet_bootstrap.cc:492] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:04.942765 6836 ts_tablet_manager.cc:1398] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:04.942927 6836 tablet_bootstrap.cc:492] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:04.943629 6840 tablet_bootstrap.cc:654] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.943823 6836 tablet_bootstrap.cc:654] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.943872 6372 tablet_service.cc:1467] Processing CreateTablet for tablet e17e4c0e86564147908d01602f82480a (DEFAULT_TABLE table=rebalance_test_table_3 [id=9bf893cef55841b394f673321b65c634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.944157 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e4c0e86564147908d01602f82480a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.944309 6107 tablet_service.cc:1467] Processing CreateTablet for tablet e17e4c0e86564147908d01602f82480a (DEFAULT_TABLE table=rebalance_test_table_3 [id=9bf893cef55841b394f673321b65c634]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.944594 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e4c0e86564147908d01602f82480a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.944756 6836 tablet_bootstrap.cc:492] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:04.944821 6836 ts_tablet_manager.cc:1367] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:04.944909 6840 tablet_bootstrap.cc:492] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:04.944981 6840 ts_tablet_manager.cc:1367] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:04.945246 6836 raft_consensus.cc:348] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.945354 6836 raft_consensus.cc:374] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.945390 6836 raft_consensus.cc:729] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.945508 6836 consensus_queue.cc:260] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.945499 6840 raft_consensus.cc:348] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.945600 6840 raft_consensus.cc:374] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.945642 6840 raft_consensus.cc:729] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.945746 6836 ts_tablet_manager.cc:1398] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.945761 6840 consensus_queue.cc:260] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.946027 6840 ts_tablet_manager.cc:1398] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.946295 6843 tablet_bootstrap.cc:492] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:04.946817 5974 tablet_service.cc:1467] Processing CreateTablet for tablet c37bf45daf71453c816e5247765220e4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9bf893cef55841b394f673321b65c634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.947057 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37bf45daf71453c816e5247765220e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.947081 6844 tablet_bootstrap.cc:492] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:04.947516 6843 tablet_bootstrap.cc:654] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.948619 6843 tablet_bootstrap.cc:492] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:04.948700 6843 ts_tablet_manager.cc:1367] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.948652 6504 tablet_service.cc:1467] Processing CreateTablet for tablet c37bf45daf71453c816e5247765220e4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=9bf893cef55841b394f673321b65c634]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:04.948894 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c37bf45daf71453c816e5247765220e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:04.949172 6843 raft_consensus.cc:348] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.949255 6844 tablet_bootstrap.cc:654] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.949285 6843 raft_consensus.cc:374] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.949333 6843 raft_consensus.cc:729] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.949458 6843 consensus_queue.cc:260] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.949717 6837 tablet_bootstrap.cc:492] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:04.950007 6843 ts_tablet_manager.cc:1398] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.950570 6837 tablet_bootstrap.cc:654] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.951448 6844 tablet_bootstrap.cc:492] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:04.951465 6838 tablet_bootstrap.cc:492] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:04.951821 6844 ts_tablet_manager.cc:1367] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:56:04.952109 6837 tablet_bootstrap.cc:492] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:04.952198 6837 ts_tablet_manager.cc:1367] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:04.952347 6838 tablet_bootstrap.cc:654] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.952579 6844 raft_consensus.cc:348] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.952690 6844 raft_consensus.cc:374] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.952725 6844 raft_consensus.cc:729] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.952672 6837 raft_consensus.cc:348] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:04.952791 6837 raft_consensus.cc:374] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.952839 6837 raft_consensus.cc:729] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.952881 6844 consensus_queue.cc:260] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.953006 6837 consensus_queue.cc:260] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:04.953167 6600 raft_consensus.cc:482] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.953258 6600 raft_consensus.cc:504] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:04.953673 6838 tablet_bootstrap.cc:492] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:04.953754 6838 ts_tablet_manager.cc:1367] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:04.954185 6838 raft_consensus.cc:348] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:04.954289 6838 raft_consensus.cc:374] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.954267 6600 leader_election.cc:288] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:04.954337 6838 raft_consensus.cc:729] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.954447 6838 consensus_queue.cc:260] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:04.954711 6838 ts_tablet_manager.cc:1398] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:04.954749 6837 ts_tablet_manager.cc:1398] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:04.954874 6838 tablet_bootstrap.cc:492] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:04.954913 6837 tablet_bootstrap.cc:492] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:04.955382 6844 ts_tablet_manager.cc:1398] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:04.955700 6838 tablet_bootstrap.cc:654] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.955746 6837 tablet_bootstrap.cc:654] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:04.957341 6837 tablet_bootstrap.cc:492] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:04.957435 6837 ts_tablet_manager.cc:1367] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:04.957855 6837 raft_consensus.cc:348] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.957949 6837 raft_consensus.cc:374] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.957975 6837 raft_consensus.cc:729] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.958071 6837 consensus_queue.cc:260] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.958281 6837 ts_tablet_manager.cc:1398] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:04.958465 6838 tablet_bootstrap.cc:492] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:04.958540 6838 ts_tablet_manager.cc:1367] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:04.960278 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc7c03eb2f447eb9d268b4cc7036dc3" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:04.960318 6838 raft_consensus.cc:348] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.960394 5994 raft_consensus.cc:2454] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:04.960430 6838 raft_consensus.cc:374] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:04.960464 6838 raft_consensus.cc:729] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:04.960602 6838 consensus_queue.cc:260] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:04.960726 6459 leader_election.cc:302] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:04.960950 6600 raft_consensus.cc:2781] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.960956 6838 ts_tablet_manager.cc:1398] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:04.960999 6600 raft_consensus.cc:482] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.961066 6600 raft_consensus.cc:3037] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.961817 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc7c03eb2f447eb9d268b4cc7036dc3" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:04.961946 6260 raft_consensus.cc:2454] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:04.962395 6600 raft_consensus.cc:504] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:04.962502 6808 raft_consensus.cc:482] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.962607 6808 raft_consensus.cc:504] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.962750 6600 leader_election.cc:288] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Requested vote from peers 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:04.962976 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc7c03eb2f447eb9d268b4cc7036dc3" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:04.963084 5994 raft_consensus.cc:3037] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.963116 6808 leader_election.cc:288] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:04.964099 5994 raft_consensus.cc:2454] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:04.964540 6459 leader_election.cc:302] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:04.964707 6600 raft_consensus.cc:2781] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.964879 6600 raft_consensus.cc:686] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 LEADER]: Becoming Leader. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Running, Role: LEADER
I20240501 19:56:04.965018 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc7c03eb2f447eb9d268b4cc7036dc3" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:04.965160 6260 raft_consensus.cc:3037] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.965209 6600 consensus_queue.cc:237] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:04.966120 6260 raft_consensus.cc:2454] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:04.966758 6608 raft_consensus.cc:482] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:04.966874 6608 raft_consensus.cc:504] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.967792 6608 leader_election.cc:288] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:04.967955 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e4c0e86564147908d01602f82480a" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:04.968096 6392 raft_consensus.cc:2454] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:04.968386 5798 leader_election.cc:302] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:04.968592 6808 raft_consensus.cc:2781] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:04.968660 6808 raft_consensus.cc:482] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:04.968703 6808 raft_consensus.cc:3037] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.969697 6808 raft_consensus.cc:504] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.970041 6808 leader_election.cc:288] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Requested vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:04.970530 5714 catalog_manager.cc:5484] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70). New cstate: current_term: 1 leader_uuid: "60b2c1ef2e86405bba90eaa1db142e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:04.971418 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e4c0e86564147908d01602f82480a" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:04.971524 6392 raft_consensus.cc:3037] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.972430 6392 raft_consensus.cc:2454] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:04.972738 5798 leader_election.cc:302] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:04.972903 6808 raft_consensus.cc:2781] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:04.972978 6808 raft_consensus.cc:686] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [term 1 LEADER]: Becoming Leader. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Running, Role: LEADER
I20240501 19:56:04.973125 6808 consensus_queue.cc:237] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:04.973836 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e4c0e86564147908d01602f82480a" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:04.974006 6392 raft_consensus.cc:2379] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d6a1d3564b3b40de9e4e4724bfa59d65 in current term 1: Already voted for candidate 4629e756401746a8bf4652f995bddb44 in this term.
I20240501 19:56:04.974535 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e4c0e86564147908d01602f82480a" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:04.974857 6063 leader_election.cc:302] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6a1d3564b3b40de9e4e4724bfa59d65; no voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975
I20240501 19:56:04.975126 6608 raft_consensus.cc:3037] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:04.975129 5714 catalog_manager.cc:5484] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4629e756401746a8bf4652f995bddb44 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "4629e756401746a8bf4652f995bddb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:04.975898 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e4c0e86564147908d01602f82480a" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:04.976068 6608 raft_consensus.cc:2726] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:04.976127 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e4c0e86564147908d01602f82480a" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:04.976365 6127 raft_consensus.cc:2454] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:05.157173 6600 raft_consensus.cc:482] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.157298 6600 raft_consensus.cc:504] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:05.157666 6600 leader_election.cc:288] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:05.158126 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37bf45daf71453c816e5247765220e4" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:05.158174 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37bf45daf71453c816e5247765220e4" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:05.158294 6260 raft_consensus.cc:2454] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:05.158310 5994 raft_consensus.cc:2454] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:05.158598 6459 leader_election.cc:302] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:05.158787 6600 raft_consensus.cc:2781] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.158843 6600 raft_consensus.cc:482] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.158879 6600 raft_consensus.cc:3037] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.159657 6600 raft_consensus.cc:504] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:05.159960 6600 leader_election.cc:288] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Requested vote from peers fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:05.160279 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37bf45daf71453c816e5247765220e4" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:05.160290 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c37bf45daf71453c816e5247765220e4" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:05.160382 5994 raft_consensus.cc:3037] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.160391 6260 raft_consensus.cc:3037] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.161124 6260 raft_consensus.cc:2454] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:05.161136 5994 raft_consensus.cc:2454] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:05.161422 6460 leader_election.cc:302] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:05.161577 6600 raft_consensus.cc:2781] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.161670 6600 raft_consensus.cc:686] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 LEADER]: Becoming Leader. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Running, Role: LEADER
I20240501 19:56:05.161837 6600 consensus_queue.cc:237] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:05.163508 5714 catalog_manager.cc:5484] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70). New cstate: current_term: 1 leader_uuid: "60b2c1ef2e86405bba90eaa1db142e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:05.286927 6863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.287127 6863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.290041 6863 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 19:56:05.361145 6600 consensus_queue.cc:1035] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.365180 6600 consensus_queue.cc:1035] T 1bc7c03eb2f447eb9d268b4cc7036dc3 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.390810 6260 raft_consensus.cc:1261] T c37bf45daf71453c816e5247765220e4 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.391079 5994 raft_consensus.cc:1261] T c37bf45daf71453c816e5247765220e4 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.391198 6894 consensus_queue.cc:1035] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.391644 6600 consensus_queue.cc:1035] T c37bf45daf71453c816e5247765220e4 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.393030 6127 raft_consensus.cc:1261] T e17e4c0e86564147908d01602f82480a P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.393395 6622 consensus_queue.cc:1035] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.399766 6392 raft_consensus.cc:1261] T e17e4c0e86564147908d01602f82480a P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.402598 6622 consensus_queue.cc:1035] T e17e4c0e86564147908d01602f82480a P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 124.093 ms
time per row: 0.0620464 ms
I20240501 19:56:05.453121 5715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:05.458137 6372 tablet_service.cc:1467] Processing CreateTablet for tablet 747ac8627fc94390b942717801d9e61f (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee101d761758444d8d5cf85a9b7bd81d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.458467 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747ac8627fc94390b942717801d9e61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.458464 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 747ac8627fc94390b942717801d9e61f (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee101d761758444d8d5cf85a9b7bd81d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.458755 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747ac8627fc94390b942717801d9e61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.459201 6239 tablet_service.cc:1467] Processing CreateTablet for tablet 3601c1f90c8f40f8ae4515bad00d412f (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee101d761758444d8d5cf85a9b7bd81d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.459478 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3601c1f90c8f40f8ae4515bad00d412f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.460796 6903 tablet_bootstrap.cc:492] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:05.461760 6504 tablet_service.cc:1467] Processing CreateTablet for tablet 3601c1f90c8f40f8ae4515bad00d412f (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee101d761758444d8d5cf85a9b7bd81d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.461747 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 3601c1f90c8f40f8ae4515bad00d412f (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee101d761758444d8d5cf85a9b7bd81d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.461865 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc35d49716049c0aaf3407d786d1100 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee101d761758444d8d5cf85a9b7bd81d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.462014 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3601c1f90c8f40f8ae4515bad00d412f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.462049 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3601c1f90c8f40f8ae4515bad00d412f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.462050 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc35d49716049c0aaf3407d786d1100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.462239 6903 tablet_bootstrap.cc:654] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.463244 6903 tablet_bootstrap.cc:492] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:05.463326 6903 ts_tablet_manager.cc:1367] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:05.463737 6903 raft_consensus.cc:348] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.463857 6903 raft_consensus.cc:374] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.463889 6903 raft_consensus.cc:729] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.463992 6903 consensus_queue.cc:260] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.464264 6903 ts_tablet_manager.cc:1398] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.464265 6107 tablet_service.cc:1467] Processing CreateTablet for tablet 747ac8627fc94390b942717801d9e61f (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee101d761758444d8d5cf85a9b7bd81d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.464406 6903 tablet_bootstrap.cc:492] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:05.464514 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 747ac8627fc94390b942717801d9e61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.464684 6905 tablet_bootstrap.cc:492] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:05.464877 6907 tablet_bootstrap.cc:492] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:05.465279 6903 tablet_bootstrap.cc:654] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.465778 6905 tablet_bootstrap.cc:654] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.465824 6907 tablet_bootstrap.cc:654] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.466431 6904 tablet_bootstrap.cc:492] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:05.466868 6903 tablet_bootstrap.cc:492] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:05.466938 6903 ts_tablet_manager.cc:1367] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:05.466972 6106 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc35d49716049c0aaf3407d786d1100 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee101d761758444d8d5cf85a9b7bd81d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.467183 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc35d49716049c0aaf3407d786d1100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.467393 6904 tablet_bootstrap.cc:654] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.467473 6903 raft_consensus.cc:348] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.467562 6903 raft_consensus.cc:374] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.467593 6903 raft_consensus.cc:729] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.467706 6903 consensus_queue.cc:260] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.467978 6903 ts_tablet_manager.cc:1398] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.468501 6904 tablet_bootstrap.cc:492] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:05.468571 6904 ts_tablet_manager.cc:1367] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.468655 6905 tablet_bootstrap.cc:492] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:05.468727 6905 ts_tablet_manager.cc:1367] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:05.468998 6904 raft_consensus.cc:348] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.469156 6904 raft_consensus.cc:374] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.469195 6904 raft_consensus.cc:729] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.469213 6905 raft_consensus.cc:348] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.469347 6905 raft_consensus.cc:374] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.469381 6905 raft_consensus.cc:729] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.469323 6904 consensus_queue.cc:260] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.469507 6905 consensus_queue.cc:260] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.469610 6904 ts_tablet_manager.cc:1398] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.469769 6905 ts_tablet_manager.cc:1398] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.470503 6907 tablet_bootstrap.cc:492] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:05.470587 6907 ts_tablet_manager.cc:1367] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:05.470728 6372 tablet_service.cc:1467] Processing CreateTablet for tablet 3bc35d49716049c0aaf3407d786d1100 (DEFAULT_TABLE table=rebalance_test_table_4 [id=ee101d761758444d8d5cf85a9b7bd81d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.470959 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3bc35d49716049c0aaf3407d786d1100. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.471010 6907 raft_consensus.cc:348] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.471094 6907 raft_consensus.cc:374] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.471123 6907 raft_consensus.cc:729] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.471248 6907 consensus_queue.cc:260] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.471504 6907 ts_tablet_manager.cc:1398] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:05.473198 6904 tablet_bootstrap.cc:492] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:05.473280 6912 tablet_bootstrap.cc:492] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:05.473999 6904 tablet_bootstrap.cc:654] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.474802 6912 tablet_bootstrap.cc:654] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.475014 6908 tablet_bootstrap.cc:492] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:05.475328 6904 tablet_bootstrap.cc:492] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:05.475394 6904 ts_tablet_manager.cc:1367] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.475718 6904 raft_consensus.cc:348] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.475806 6904 raft_consensus.cc:374] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.475805 6908 tablet_bootstrap.cc:654] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.475834 6904 raft_consensus.cc:729] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.475940 6904 consensus_queue.cc:260] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.476239 6904 ts_tablet_manager.cc:1398] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:05.476961 6908 tablet_bootstrap.cc:492] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:05.477092 6908 ts_tablet_manager.cc:1367] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:05.477244 6912 tablet_bootstrap.cc:492] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:05.477365 6912 ts_tablet_manager.cc:1367] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:05.477533 6908 raft_consensus.cc:348] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.477651 6908 raft_consensus.cc:374] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.477734 6908 raft_consensus.cc:729] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.477773 6912 raft_consensus.cc:348] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.477931 6912 raft_consensus.cc:374] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.477898 6908 consensus_queue.cc:260] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.477998 6912 raft_consensus.cc:729] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.478170 6912 consensus_queue.cc:260] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.478236 6908 ts_tablet_manager.cc:1398] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.478420 6912 ts_tablet_manager.cc:1398] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.478713 6912 tablet_bootstrap.cc:492] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:05.479537 6912 tablet_bootstrap.cc:654] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.481324 6912 tablet_bootstrap.cc:492] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:05.481488 6912 ts_tablet_manager.cc:1367] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:05.481995 6912 raft_consensus.cc:348] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.482112 6912 raft_consensus.cc:374] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.482156 6912 raft_consensus.cc:729] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.482277 6912 consensus_queue.cc:260] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.482496 6912 ts_tablet_manager.cc:1398] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:05.531352 6600 raft_consensus.cc:482] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.531455 6600 raft_consensus.cc:504] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.531857 6600 leader_election.cc:288] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:05.532223 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3601c1f90c8f40f8ae4515bad00d412f" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:05.532284 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3601c1f90c8f40f8ae4515bad00d412f" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:05.532383 5994 raft_consensus.cc:2454] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:05.532444 6260 raft_consensus.cc:2454] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:05.532728 6459 leader_election.cc:302] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:05.532894 6600 raft_consensus.cc:2781] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.532958 6600 raft_consensus.cc:482] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.533007 6600 raft_consensus.cc:3037] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.533888 6600 raft_consensus.cc:504] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.534211 6600 leader_election.cc:288] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Requested vote from peers 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:05.534536 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3601c1f90c8f40f8ae4515bad00d412f" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:05.534663 6260 raft_consensus.cc:3037] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.534641 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3601c1f90c8f40f8ae4515bad00d412f" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:05.534749 5994 raft_consensus.cc:3037] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.535606 6260 raft_consensus.cc:2454] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:05.535606 5994 raft_consensus.cc:2454] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:05.535925 6459 leader_election.cc:302] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:05.536084 6600 raft_consensus.cc:2781] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.536168 6600 raft_consensus.cc:686] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 LEADER]: Becoming Leader. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Running, Role: LEADER
I20240501 19:56:05.536337 6600 consensus_queue.cc:237] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.537874 5717 catalog_manager.cc:5484] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70). New cstate: current_term: 1 leader_uuid: "60b2c1ef2e86405bba90eaa1db142e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:05.591218 6622 raft_consensus.cc:482] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.591377 6622 raft_consensus.cc:504] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.591851 6622 leader_election.cc:288] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:05.592253 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc35d49716049c0aaf3407d786d1100" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:05.592285 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc35d49716049c0aaf3407d786d1100" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:05.592415 6127 raft_consensus.cc:2454] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:05.592404 6392 raft_consensus.cc:2454] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:05.592720 5798 leader_election.cc:302] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:05.592981 6622 raft_consensus.cc:2781] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.593078 6622 raft_consensus.cc:482] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.593135 6622 raft_consensus.cc:3037] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.593945 6622 raft_consensus.cc:504] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.594235 6622 leader_election.cc:288] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Requested vote from peers a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:05.594635 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc35d49716049c0aaf3407d786d1100" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:05.594658 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3bc35d49716049c0aaf3407d786d1100" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:05.594753 6392 raft_consensus.cc:3037] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.594759 6127 raft_consensus.cc:3037] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.595487 6127 raft_consensus.cc:2454] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:05.595515 6392 raft_consensus.cc:2454] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:05.595757 5798 leader_election.cc:302] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:05.595949 6622 raft_consensus.cc:2781] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.596031 6622 raft_consensus.cc:686] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [term 1 LEADER]: Becoming Leader. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Running, Role: LEADER
I20240501 19:56:05.596170 6622 consensus_queue.cc:237] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.597824 5717 catalog_manager.cc:5484] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4629e756401746a8bf4652f995bddb44 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "4629e756401746a8bf4652f995bddb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:05.639792 6900 raft_consensus.cc:482] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:05.639923 6900 raft_consensus.cc:504] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.640345 6900 leader_election.cc:288] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:05.640806 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ac8627fc94390b942717801d9e61f" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:05.640812 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ac8627fc94390b942717801d9e61f" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:05.640940 5862 raft_consensus.cc:2454] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:05.640941 6127 raft_consensus.cc:2454] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:05.641259 6328 leader_election.cc:302] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:05.641446 6900 raft_consensus.cc:2781] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:05.641508 6900 raft_consensus.cc:482] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:05.641534 6900 raft_consensus.cc:3037] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.642333 6900 raft_consensus.cc:504] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.642647 6900 leader_election.cc:288] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Requested vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:05.642930 5862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ac8627fc94390b942717801d9e61f" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44"
I20240501 19:56:05.642973 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "747ac8627fc94390b942717801d9e61f" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:05.643033 5862 raft_consensus.cc:3037] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.643072 6127 raft_consensus.cc:3037] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:05.643785 5862 raft_consensus.cc:2454] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:05.643834 6127 raft_consensus.cc:2454] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:05.644029 6328 leader_election.cc:302] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:05.644220 6900 raft_consensus.cc:2781] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:05.644315 6900 raft_consensus.cc:686] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [term 1 LEADER]: Becoming Leader. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Running, Role: LEADER
I20240501 19:56:05.644454 6900 consensus_queue.cc:237] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:05.646118 5717 catalog_manager.cc:5484] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69). New cstate: current_term: 1 leader_uuid: "a2a04e6cd48541839ce83dc1d1790975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:05.762409 6921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:05.762627 6921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:05.765590 6921 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 19:56:05.857664 6127 raft_consensus.cc:1261] T 3bc35d49716049c0aaf3407d786d1100 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.858270 6622 consensus_queue.cc:1035] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.859663 6392 raft_consensus.cc:1261] T 3bc35d49716049c0aaf3407d786d1100 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.860131 6622 consensus_queue.cc:1035] T 3bc35d49716049c0aaf3407d786d1100 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.861773 6260 raft_consensus.cc:1261] T 3601c1f90c8f40f8ae4515bad00d412f P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.862112 5994 raft_consensus.cc:1261] T 3601c1f90c8f40f8ae4515bad00d412f P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.862522 6600 consensus_queue.cc:1035] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.863031 6600 consensus_queue.cc:1035] T 3601c1f90c8f40f8ae4515bad00d412f P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.870981 5862 raft_consensus.cc:1261] T 747ac8627fc94390b942717801d9e61f P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.870998 6126 raft_consensus.cc:1261] T 747ac8627fc94390b942717801d9e61f P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:05.871568 6822 consensus_queue.cc:1035] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:05.871937 6900 consensus_queue.cc:1035] T 747ac8627fc94390b942717801d9e61f P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 125.581 ms
time per row: 0.0627903 ms
I20240501 19:56:05.931953 5719 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:05.937026 6239 tablet_service.cc:1467] Processing CreateTablet for tablet 46e06196fcff476fb8e75fae159a6b42 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5dd210e4092d468e90fcf73415c7512b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.937384 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e06196fcff476fb8e75fae159a6b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.937361 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 46e06196fcff476fb8e75fae159a6b42 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5dd210e4092d468e90fcf73415c7512b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.937606 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e06196fcff476fb8e75fae159a6b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.937938 6504 tablet_service.cc:1467] Processing CreateTablet for tablet 46e06196fcff476fb8e75fae159a6b42 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5dd210e4092d468e90fcf73415c7512b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.938175 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e06196fcff476fb8e75fae159a6b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.939508 6905 tablet_bootstrap.cc:492] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:05.939647 6908 tablet_bootstrap.cc:492] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:05.939730 6503 tablet_service.cc:1467] Processing CreateTablet for tablet 77ce1fa3ae5942e08cac2dad70319167 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5dd210e4092d468e90fcf73415c7512b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.939976 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ce1fa3ae5942e08cac2dad70319167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.940155 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdb29bb3e4b4cf1ad2848bac08c83d0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5dd210e4092d468e90fcf73415c7512b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.940372 6905 tablet_bootstrap.cc:654] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.940372 6908 tablet_bootstrap.cc:654] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.940424 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdb29bb3e4b4cf1ad2848bac08c83d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.941015 6907 tablet_bootstrap.cc:492] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:05.941808 6908 tablet_bootstrap.cc:492] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:05.941884 6907 tablet_bootstrap.cc:654] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.941895 6908 ts_tablet_manager.cc:1367] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:05.942068 6372 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdb29bb3e4b4cf1ad2848bac08c83d0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5dd210e4092d468e90fcf73415c7512b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.942322 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdb29bb3e4b4cf1ad2848bac08c83d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.942353 6908 raft_consensus.cc:348] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.942456 6908 raft_consensus.cc:374] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.942490 6908 raft_consensus.cc:729] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.942610 6908 consensus_queue.cc:260] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.942895 6908 ts_tablet_manager.cc:1398] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:05.943044 6908 tablet_bootstrap.cc:492] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:05.943663 6239 tablet_service.cc:1467] Processing CreateTablet for tablet 77ce1fa3ae5942e08cac2dad70319167 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5dd210e4092d468e90fcf73415c7512b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.943789 6908 tablet_bootstrap.cc:654] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.943915 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ce1fa3ae5942e08cac2dad70319167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.944628 6106 tablet_service.cc:1467] Processing CreateTablet for tablet 8bdb29bb3e4b4cf1ad2848bac08c83d0 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5dd210e4092d468e90fcf73415c7512b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.944707 6908 tablet_bootstrap.cc:492] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:05.944769 6903 tablet_bootstrap.cc:492] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:05.944773 6908 ts_tablet_manager.cc:1367] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:05.944913 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bdb29bb3e4b4cf1ad2848bac08c83d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.945156 6904 tablet_bootstrap.cc:492] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:05.945183 6908 raft_consensus.cc:348] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:05.945258 6908 raft_consensus.cc:374] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.945286 6908 raft_consensus.cc:729] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.945969 6904 tablet_bootstrap.cc:654] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.946620 6907 tablet_bootstrap.cc:492] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:05.946599 6908 consensus_queue.cc:260] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:05.946699 6907 ts_tablet_manager.cc:1367] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20240501 19:56:05.946938 6903 tablet_bootstrap.cc:654] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.946945 6912 tablet_bootstrap.cc:492] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:05.946950 6908 ts_tablet_manager.cc:1398] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.947093 6907 raft_consensus.cc:348] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.947183 6907 raft_consensus.cc:374] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.947213 6907 raft_consensus.cc:729] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.947324 6907 consensus_queue.cc:260] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.947566 6907 ts_tablet_manager.cc:1398] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:05.947572 6904 tablet_bootstrap.cc:492] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:05.947640 6904 ts_tablet_manager.cc:1367] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:05.947717 6912 tablet_bootstrap.cc:654] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.948061 6904 raft_consensus.cc:348] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:05.948154 6904 raft_consensus.cc:374] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.948182 6904 raft_consensus.cc:729] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.948338 6903 tablet_bootstrap.cc:492] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:05.948300 6904 consensus_queue.cc:260] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:05.948401 6903 ts_tablet_manager.cc:1367] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:05.948551 6904 ts_tablet_manager.cc:1398] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.948779 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 77ce1fa3ae5942e08cac2dad70319167 (DEFAULT_TABLE table=rebalance_test_table_5 [id=5dd210e4092d468e90fcf73415c7512b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:05.948794 6903 raft_consensus.cc:348] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:05.948892 6903 raft_consensus.cc:374] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.948927 6903 raft_consensus.cc:729] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.948980 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77ce1fa3ae5942e08cac2dad70319167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:05.949522 6903 consensus_queue.cc:260] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:05.949815 6903 ts_tablet_manager.cc:1398] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.951828 6907 tablet_bootstrap.cc:492] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:05.952667 6907 tablet_bootstrap.cc:654] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.952777 6912 tablet_bootstrap.cc:492] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:05.952863 6912 ts_tablet_manager.cc:1367] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:05.953094 6905 tablet_bootstrap.cc:492] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:05.953176 6905 ts_tablet_manager.cc:1367] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240501 19:56:05.953318 6912 raft_consensus.cc:348] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:05.953435 6912 raft_consensus.cc:374] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.953469 6912 raft_consensus.cc:729] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.953573 6912 consensus_queue.cc:260] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:05.953614 6905 raft_consensus.cc:348] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.953667 6907 tablet_bootstrap.cc:492] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:05.953701 6905 raft_consensus.cc:374] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.953739 6907 ts_tablet_manager.cc:1367] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.953750 6905 raft_consensus.cc:729] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.953855 6912 ts_tablet_manager.cc:1398] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.953883 6905 consensus_queue.cc:260] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:05.954195 6905 ts_tablet_manager.cc:1398] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:05.954358 6905 tablet_bootstrap.cc:492] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:05.954785 6907 raft_consensus.cc:348] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:05.954869 6907 raft_consensus.cc:374] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.954905 6907 raft_consensus.cc:729] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.955018 6907 consensus_queue.cc:260] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:05.955231 6905 tablet_bootstrap.cc:654] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:05.955539 6907 ts_tablet_manager.cc:1398] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.956532 6905 tablet_bootstrap.cc:492] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:05.956609 6905 ts_tablet_manager.cc:1367] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:05.956986 6905 raft_consensus.cc:348] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:05.957132 6905 raft_consensus.cc:374] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:05.957177 6905 raft_consensus.cc:729] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:05.957283 6905 consensus_queue.cc:260] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:05.957481 6905 ts_tablet_manager.cc:1398] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:06.102566 6622 raft_consensus.cc:482] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:06.102689 6622 raft_consensus.cc:504] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:06.103062 6622 leader_election.cc:288] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:06.103489 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdb29bb3e4b4cf1ad2848bac08c83d0" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:06.103489 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdb29bb3e4b4cf1ad2848bac08c83d0" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:06.103636 6127 raft_consensus.cc:2454] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:06.103636 6392 raft_consensus.cc:2454] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:06.103940 5798 leader_election.cc:302] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:06.104123 6622 raft_consensus.cc:2781] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:06.104188 6622 raft_consensus.cc:482] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:06.104240 6622 raft_consensus.cc:3037] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.105026 6622 raft_consensus.cc:504] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:06.105343 6622 leader_election.cc:288] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Requested vote from peers a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:06.105680 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdb29bb3e4b4cf1ad2848bac08c83d0" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:06.105700 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bdb29bb3e4b4cf1ad2848bac08c83d0" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:06.105818 6392 raft_consensus.cc:3037] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.105872 6127 raft_consensus.cc:3037] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.106745 6392 raft_consensus.cc:2454] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:06.106745 6127 raft_consensus.cc:2454] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:06.107029 5798 leader_election.cc:302] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:06.107168 6622 raft_consensus.cc:2781] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:06.107275 6622 raft_consensus.cc:686] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [term 1 LEADER]: Becoming Leader. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Running, Role: LEADER
I20240501 19:56:06.107417 6622 consensus_queue.cc:237] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:06.109005 5711 catalog_manager.cc:5484] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4629e756401746a8bf4652f995bddb44 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "4629e756401746a8bf4652f995bddb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:06.118809 6626 raft_consensus.cc:482] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:06.118921 6626 raft_consensus.cc:504] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:06.119259 6626 leader_election.cc:288] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:06.119673 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce1fa3ae5942e08cac2dad70319167" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:06.119818 6523 raft_consensus.cc:2454] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:06.119715 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce1fa3ae5942e08cac2dad70319167" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:06.119959 6260 raft_consensus.cc:2454] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:06.120146 5930 leader_election.cc:302] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:06.120326 6626 raft_consensus.cc:2781] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:06.120395 6626 raft_consensus.cc:482] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:06.120431 6626 raft_consensus.cc:3037] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.121295 6626 raft_consensus.cc:504] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:06.121594 6626 leader_election.cc:288] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:06.121919 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce1fa3ae5942e08cac2dad70319167" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:06.121920 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77ce1fa3ae5942e08cac2dad70319167" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:06.122035 6523 raft_consensus.cc:3037] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.122035 6260 raft_consensus.cc:3037] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.122988 6523 raft_consensus.cc:2454] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:06.123001 6260 raft_consensus.cc:2454] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:06.123270 5930 leader_election.cc:302] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:06.123490 6626 raft_consensus.cc:2781] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:06.123643 6626 raft_consensus.cc:686] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [term 1 LEADER]: Becoming Leader. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Running, Role: LEADER
I20240501 19:56:06.123814 6626 consensus_queue.cc:237] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:06.125486 5711 catalog_manager.cc:5484] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3489b69a0428491e957e46effb9b1146 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "3489b69a0428491e957e46effb9b1146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:06.146526 6600 raft_consensus.cc:482] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:06.146643 6600 raft_consensus.cc:504] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.146984 6600 leader_election.cc:288] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:06.147374 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e06196fcff476fb8e75fae159a6b42" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:06.147383 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e06196fcff476fb8e75fae159a6b42" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:06.147501 5994 raft_consensus.cc:2454] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:06.147508 6260 raft_consensus.cc:2454] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:06.147768 6460 leader_election.cc:302] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:06.147959 6600 raft_consensus.cc:2781] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:06.148025 6600 raft_consensus.cc:482] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:06.148068 6600 raft_consensus.cc:3037] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.148838 6600 raft_consensus.cc:504] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.149185 6600 leader_election.cc:288] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Requested vote from peers fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:06.149489 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e06196fcff476fb8e75fae159a6b42" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:06.149507 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e06196fcff476fb8e75fae159a6b42" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:06.149601 6260 raft_consensus.cc:3037] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.149602 5994 raft_consensus.cc:3037] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.150477 5994 raft_consensus.cc:2454] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:06.150477 6260 raft_consensus.cc:2454] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:06.150746 6460 leader_election.cc:302] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:06.150910 6600 raft_consensus.cc:2781] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:06.150998 6600 raft_consensus.cc:686] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 LEADER]: Becoming Leader. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Running, Role: LEADER
I20240501 19:56:06.151137 6600 consensus_queue.cc:237] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.152709 5711 catalog_manager.cc:5484] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70). New cstate: current_term: 1 leader_uuid: "60b2c1ef2e86405bba90eaa1db142e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:06.284476 6979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:06.284688 6979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:06.287683 6979 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 19:56:06.358518 6127 raft_consensus.cc:1261] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:06.358963 6622 consensus_queue.cc:1035] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:06.362769 6260 raft_consensus.cc:1261] T 77ce1fa3ae5942e08cac2dad70319167 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:06.363184 6626 consensus_queue.cc:1035] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:06.363783 6260 raft_consensus.cc:1261] T 46e06196fcff476fb8e75fae159a6b42 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:06.364166 6600 consensus_queue.cc:1035] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:06.365502 6523 raft_consensus.cc:1261] T 77ce1fa3ae5942e08cac2dad70319167 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:06.366798 5995 raft_consensus.cc:1261] T 46e06196fcff476fb8e75fae159a6b42 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:06.368304 6894 consensus_queue.cc:1035] T 46e06196fcff476fb8e75fae159a6b42 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:06.368960 6976 consensus_queue.cc:1035] T 77ce1fa3ae5942e08cac2dad70319167 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:06.386104 6392 raft_consensus.cc:1261] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:06.387319 6622 consensus_queue.cc:1035] T 8bdb29bb3e4b4cf1ad2848bac08c83d0 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 100.515 ms
time per row: 0.0502575 ms
I20240501 19:56:06.430508 5711 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:06.436201 6106 tablet_service.cc:1467] Processing CreateTablet for tablet 07fd0c6dc02740469cfa36ee5ef587a0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=03d935e42ad1469db2a4af5987f58581]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:06.436538 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fd0c6dc02740469cfa36ee5ef587a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.436669 6239 tablet_service.cc:1467] Processing CreateTablet for tablet a1aa22f900e54f1dbdc735b7aa2377e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=03d935e42ad1469db2a4af5987f58581]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:06.436933 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1aa22f900e54f1dbdc735b7aa2377e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.436906 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 07fd0c6dc02740469cfa36ee5ef587a0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=03d935e42ad1469db2a4af5987f58581]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:06.437166 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 0609d499c8d3425697918f47a660d403 (DEFAULT_TABLE table=rebalance_test_table_6 [id=03d935e42ad1469db2a4af5987f58581]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:06.437196 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fd0c6dc02740469cfa36ee5ef587a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.437413 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0609d499c8d3425697918f47a660d403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.439160 6905 tablet_bootstrap.cc:492] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:06.439282 6903 tablet_bootstrap.cc:492] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:06.439316 6912 tablet_bootstrap.cc:492] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:06.439925 6903 tablet_bootstrap.cc:654] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:06.439983 6905 tablet_bootstrap.cc:654] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:06.440093 6912 tablet_bootstrap.cc:654] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:06.440850 5974 tablet_service.cc:1467] Processing CreateTablet for tablet a1aa22f900e54f1dbdc735b7aa2377e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=03d935e42ad1469db2a4af5987f58581]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:06.441197 6371 tablet_service.cc:1467] Processing CreateTablet for tablet 0609d499c8d3425697918f47a660d403 (DEFAULT_TABLE table=rebalance_test_table_6 [id=03d935e42ad1469db2a4af5987f58581]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:06.441442 6371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0609d499c8d3425697918f47a660d403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.441396 6503 tablet_service.cc:1467] Processing CreateTablet for tablet a1aa22f900e54f1dbdc735b7aa2377e6 (DEFAULT_TABLE table=rebalance_test_table_6 [id=03d935e42ad1469db2a4af5987f58581]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:06.441437 6106 tablet_service.cc:1467] Processing CreateTablet for tablet 0609d499c8d3425697918f47a660d403 (DEFAULT_TABLE table=rebalance_test_table_6 [id=03d935e42ad1469db2a4af5987f58581]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:06.441604 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0609d499c8d3425697918f47a660d403. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.441630 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1aa22f900e54f1dbdc735b7aa2377e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.443446 6904 tablet_bootstrap.cc:492] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:06.443763 6372 tablet_service.cc:1467] Processing CreateTablet for tablet 07fd0c6dc02740469cfa36ee5ef587a0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=03d935e42ad1469db2a4af5987f58581]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:06.444007 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07fd0c6dc02740469cfa36ee5ef587a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.444278 6904 tablet_bootstrap.cc:654] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:06.444399 6905 tablet_bootstrap.cc:492] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:06.444485 6905 ts_tablet_manager.cc:1367] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:06.444909 6905 raft_consensus.cc:348] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.445024 6905 raft_consensus.cc:374] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:06.445082 6905 raft_consensus.cc:729] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.445206 6904 tablet_bootstrap.cc:492] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:06.445199 6905 consensus_queue.cc:260] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.445267 6904 ts_tablet_manager.cc:1367] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:06.445392 6908 tablet_bootstrap.cc:492] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:06.445475 6905 ts_tablet_manager.cc:1398] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:06.445650 6904 raft_consensus.cc:348] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:06.445762 6904 raft_consensus.cc:374] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:06.445796 6904 raft_consensus.cc:729] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.445914 6904 consensus_queue.cc:260] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:06.446161 6908 tablet_bootstrap.cc:654] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:06.446178 6904 ts_tablet_manager.cc:1398] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:06.447320 6908 tablet_bootstrap.cc:492] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:06.447417 6908 ts_tablet_manager.cc:1367] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:06.447641 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1aa22f900e54f1dbdc735b7aa2377e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:06.447847 6908 raft_consensus.cc:348] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.447952 6908 raft_consensus.cc:374] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:06.447983 6908 raft_consensus.cc:729] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.448091 6908 consensus_queue.cc:260] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.448330 6908 ts_tablet_manager.cc:1398] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:06.449793 6907 tablet_bootstrap.cc:492] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:06.450030 6904 tablet_bootstrap.cc:492] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:06.450614 6903 tablet_bootstrap.cc:492] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:06.450626 6907 tablet_bootstrap.cc:654] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:06.450690 6903 ts_tablet_manager.cc:1367] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:56:06.450947 6904 tablet_bootstrap.cc:654] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:06.451164 6903 raft_consensus.cc:348] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:06.451264 6903 raft_consensus.cc:374] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:06.451301 6903 raft_consensus.cc:729] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.451424 6903 consensus_queue.cc:260] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:06.451965 6903 ts_tablet_manager.cc:1398] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:06.451987 6907 tablet_bootstrap.cc:492] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:06.452050 6907 ts_tablet_manager.cc:1367] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:06.452153 6912 tablet_bootstrap.cc:492] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:06.452219 6912 ts_tablet_manager.cc:1367] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20240501 19:56:06.452349 6903 tablet_bootstrap.cc:492] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:06.452440 6907 raft_consensus.cc:348] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.452543 6907 raft_consensus.cc:374] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:06.452574 6907 raft_consensus.cc:729] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.452601 6912 raft_consensus.cc:348] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:06.452710 6912 raft_consensus.cc:374] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:06.452754 6912 raft_consensus.cc:729] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.452703 6907 consensus_queue.cc:260] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.452863 6912 consensus_queue.cc:260] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:06.452966 6907 ts_tablet_manager.cc:1398] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:06.453135 6912 ts_tablet_manager.cc:1398] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:06.453286 6903 tablet_bootstrap.cc:654] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:06.454352 6904 tablet_bootstrap.cc:492] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:06.454439 6904 ts_tablet_manager.cc:1367] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:06.454888 6903 tablet_bootstrap.cc:492] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:06.454962 6903 ts_tablet_manager.cc:1367] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:06.455255 6904 raft_consensus.cc:348] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:06.455349 6904 raft_consensus.cc:374] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:06.455317 6903 raft_consensus.cc:348] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:06.455379 6904 raft_consensus.cc:729] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.455413 6903 raft_consensus.cc:374] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:06.455446 6903 raft_consensus.cc:729] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.455535 6904 consensus_queue.cc:260] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:06.455546 6903 consensus_queue.cc:260] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:06.455771 6912 tablet_bootstrap.cc:492] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:06.455782 6903 ts_tablet_manager.cc:1398] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:06.456190 6904 ts_tablet_manager.cc:1398] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:06.456586 6912 tablet_bootstrap.cc:654] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:06.457937 6912 tablet_bootstrap.cc:492] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:06.458021 6912 ts_tablet_manager.cc:1367] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:06.458424 6912 raft_consensus.cc:348] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:06.458510 6912 raft_consensus.cc:374] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:06.458544 6912 raft_consensus.cc:729] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:06.459476 6912 consensus_queue.cc:260] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:06.459846 6912 ts_tablet_manager.cc:1398] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:06.552127 6900 raft_consensus.cc:482] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:06.552246 6900 raft_consensus.cc:504] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:06.552660 6900 leader_election.cc:288] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:06.553097 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0609d499c8d3425697918f47a660d403" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:06.553110 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0609d499c8d3425697918f47a660d403" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:06.553234 5863 raft_consensus.cc:2454] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:06.553236 6127 raft_consensus.cc:2454] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:06.553550 6327 leader_election.cc:302] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:06.553735 6900 raft_consensus.cc:2781] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:06.553807 6900 raft_consensus.cc:482] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:06.553834 6900 raft_consensus.cc:3037] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.554626 6900 raft_consensus.cc:504] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:06.554937 6900 leader_election.cc:288] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Requested vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:06.555258 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0609d499c8d3425697918f47a660d403" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44"
I20240501 19:56:06.555259 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0609d499c8d3425697918f47a660d403" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:06.555373 5863 raft_consensus.cc:3037] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.555374 6127 raft_consensus.cc:3037] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.556277 5863 raft_consensus.cc:2454] T 0609d499c8d3425697918f47a660d403 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:06.556284 6127 raft_consensus.cc:2454] T 0609d499c8d3425697918f47a660d403 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:06.556560 6328 leader_election.cc:302] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:06.556707 6900 raft_consensus.cc:2781] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:06.556814 6900 raft_consensus.cc:686] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [term 1 LEADER]: Becoming Leader. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Running, Role: LEADER
I20240501 19:56:06.556967 6900 consensus_queue.cc:237] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:06.558640 5711 catalog_manager.cc:5484] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69). New cstate: current_term: 1 leader_uuid: "a2a04e6cd48541839ce83dc1d1790975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:06.718840 6753 raft_consensus.cc:482] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:06.718963 6753 raft_consensus.cc:504] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.719412 6753 leader_election.cc:288] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835), 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011)
I20240501 19:56:06.719800 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aa22f900e54f1dbdc735b7aa2377e6" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:06.719854 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aa22f900e54f1dbdc735b7aa2377e6" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:06.719938 5994 raft_consensus.cc:2454] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:06.720010 6523 raft_consensus.cc:2454] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:06.720289 6195 leader_election.cc:302] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:06.720484 6753 raft_consensus.cc:2781] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:06.720553 6753 raft_consensus.cc:482] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:06.720580 6753 raft_consensus.cc:3037] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.721546 6753 raft_consensus.cc:504] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.721837 6753 leader_election.cc:288] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Requested vote from peers 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835), 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011)
I20240501 19:56:06.722180 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aa22f900e54f1dbdc735b7aa2377e6" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:06.722244 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1aa22f900e54f1dbdc735b7aa2377e6" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:06.722288 5994 raft_consensus.cc:3037] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.722368 6523 raft_consensus.cc:3037] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.723168 5994 raft_consensus.cc:2454] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:06.723208 6523 raft_consensus.cc:2454] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:06.723531 6195 leader_election.cc:302] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:06.723795 6753 raft_consensus.cc:2781] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:06.723907 6753 raft_consensus.cc:686] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [term 1 LEADER]: Becoming Leader. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Running, Role: LEADER
I20240501 19:56:06.724041 6753 consensus_queue.cc:237] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:06.725699 5711 catalog_manager.cc:5484] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1af421bf6a480587c69697af9f39dd (127.5.139.68). New cstate: current_term: 1 leader_uuid: "fb1af421bf6a480587c69697af9f39dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:06.864523 6622 raft_consensus.cc:482] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:06.864655 6622 raft_consensus.cc:504] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:06.865069 6622 leader_election.cc:288] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:06.865434 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fd0c6dc02740469cfa36ee5ef587a0" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:06.865576 6127 raft_consensus.cc:2454] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:06.865906 5798 leader_election.cc:302] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:06.866109 6622 raft_consensus.cc:2781] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:06.865578 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fd0c6dc02740469cfa36ee5ef587a0" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:06.866166 6622 raft_consensus.cc:482] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:06.866192 6622 raft_consensus.cc:3037] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.866235 6392 raft_consensus.cc:2454] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:06.867084 6622 raft_consensus.cc:504] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:06.867372 6622 leader_election.cc:288] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Requested vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:06.867738 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fd0c6dc02740469cfa36ee5ef587a0" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:06.867875 6127 raft_consensus.cc:3037] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.867869 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07fd0c6dc02740469cfa36ee5ef587a0" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:06.867970 6392 raft_consensus.cc:3037] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:06.868649 6127 raft_consensus.cc:2454] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:06.868777 6392 raft_consensus.cc:2454] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:06.868921 5798 leader_election.cc:302] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:06.869113 6622 raft_consensus.cc:2781] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:06.869204 6622 raft_consensus.cc:686] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [term 1 LEADER]: Becoming Leader. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Running, Role: LEADER
I20240501 19:56:06.869354 6622 consensus_queue.cc:237] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:06.871047 5711 catalog_manager.cc:5484] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4629e756401746a8bf4652f995bddb44 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "4629e756401746a8bf4652f995bddb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:06.948889 6953 consensus_queue.cc:1035] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:06.951946 7040 consensus_queue.cc:1035] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:07.004972 7037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:07.005239 7037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:07.008199 7037 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 19:56:07.089852 5994 raft_consensus.cc:1261] T a1aa22f900e54f1dbdc735b7aa2377e6 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.089970 6523 raft_consensus.cc:1261] T a1aa22f900e54f1dbdc735b7aa2377e6 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.090814 7011 consensus_queue.cc:1035] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.091140 7011 consensus_queue.cc:1035] T a1aa22f900e54f1dbdc735b7aa2377e6 P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.091984 6392 raft_consensus.cc:1261] T 07fd0c6dc02740469cfa36ee5ef587a0 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.092109 6126 raft_consensus.cc:1261] T 07fd0c6dc02740469cfa36ee5ef587a0 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.092571 6622 consensus_queue.cc:1035] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.095319 6622 consensus_queue.cc:1035] T 07fd0c6dc02740469cfa36ee5ef587a0 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.765 ms
time per row: 0.0533825 ms
I20240501 19:56:07.156841 5714 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:07.161803 6503 tablet_service.cc:1467] Processing CreateTablet for tablet 8d61f765fc274f2d841c0843ec8cc1fa (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbf4c4ba9ed046daa2b952952ca5d1a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.161919 6239 tablet_service.cc:1467] Processing CreateTablet for tablet 8d61f765fc274f2d841c0843ec8cc1fa (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbf4c4ba9ed046daa2b952952ca5d1a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.162111 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d61f765fc274f2d841c0843ec8cc1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.162199 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d61f765fc274f2d841c0843ec8cc1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.163983 7102 tablet_bootstrap.cc:492] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:07.164145 6504 tablet_service.cc:1467] Processing CreateTablet for tablet ee048890b02446b1a7cdd28b3a54e223 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbf4c4ba9ed046daa2b952952ca5d1a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.164335 7103 tablet_bootstrap.cc:492] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:07.164366 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee048890b02446b1a7cdd28b3a54e223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.164520 6372 tablet_service.cc:1467] Processing CreateTablet for tablet f41918d1c0094ec9aa4b6d1b8c57b9d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbf4c4ba9ed046daa2b952952ca5d1a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.164762 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f41918d1c0094ec9aa4b6d1b8c57b9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.165004 7102 tablet_bootstrap.cc:654] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.165856 7103 tablet_bootstrap.cc:654] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.166131 6106 tablet_service.cc:1467] Processing CreateTablet for tablet f41918d1c0094ec9aa4b6d1b8c57b9d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbf4c4ba9ed046daa2b952952ca5d1a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.166386 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f41918d1c0094ec9aa4b6d1b8c57b9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.167142 7106 tablet_bootstrap.cc:492] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:07.167163 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 8d61f765fc274f2d841c0843ec8cc1fa (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbf4c4ba9ed046daa2b952952ca5d1a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.167419 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d61f765fc274f2d841c0843ec8cc1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.169239 7106 tablet_bootstrap.cc:654] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.169266 5975 tablet_service.cc:1467] Processing CreateTablet for tablet ee048890b02446b1a7cdd28b3a54e223 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbf4c4ba9ed046daa2b952952ca5d1a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.169498 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee048890b02446b1a7cdd28b3a54e223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.169937 7107 tablet_bootstrap.cc:492] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:07.169978 7102 tablet_bootstrap.cc:492] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:07.170055 7102 ts_tablet_manager.cc:1367] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:07.170204 5843 tablet_service.cc:1467] Processing CreateTablet for tablet f41918d1c0094ec9aa4b6d1b8c57b9d2 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbf4c4ba9ed046daa2b952952ca5d1a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.170460 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f41918d1c0094ec9aa4b6d1b8c57b9d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.170491 7102 raft_consensus.cc:348] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:07.170581 7102 raft_consensus.cc:374] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.170610 7102 raft_consensus.cc:729] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.170730 7102 consensus_queue.cc:260] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:07.170818 7107 tablet_bootstrap.cc:654] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.170997 7102 ts_tablet_manager.cc:1398] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.172008 7110 tablet_bootstrap.cc:492] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:07.172859 6239 tablet_service.cc:1467] Processing CreateTablet for tablet ee048890b02446b1a7cdd28b3a54e223 (DEFAULT_TABLE table=rebalance_test_table_7 [id=dbf4c4ba9ed046daa2b952952ca5d1a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.172924 7103 tablet_bootstrap.cc:492] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:07.173000 7103 ts_tablet_manager.cc:1367] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:56:07.173116 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee048890b02446b1a7cdd28b3a54e223. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.173116 7110 tablet_bootstrap.cc:654] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.173384 7103 raft_consensus.cc:348] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:07.173472 7103 raft_consensus.cc:374] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.173507 7103 raft_consensus.cc:729] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.173638 7103 consensus_queue.cc:260] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:07.173897 7103 ts_tablet_manager.cc:1398] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.174474 7103 tablet_bootstrap.cc:492] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:07.175310 7103 tablet_bootstrap.cc:654] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.176177 7107 tablet_bootstrap.cc:492] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:07.176266 7110 tablet_bootstrap.cc:492] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:07.176266 7107 ts_tablet_manager.cc:1367] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:07.176338 7110 ts_tablet_manager.cc:1367] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:07.176714 7110 raft_consensus.cc:348] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:07.176710 7107 raft_consensus.cc:348] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:07.176810 7110 raft_consensus.cc:374] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.176810 7107 raft_consensus.cc:374] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.176844 7110 raft_consensus.cc:729] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.176844 7107 raft_consensus.cc:729] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.177019 7107 consensus_queue.cc:260] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:07.176971 7110 consensus_queue.cc:260] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:07.177328 7110 ts_tablet_manager.cc:1398] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.177337 7102 tablet_bootstrap.cc:492] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:07.177479 7110 tablet_bootstrap.cc:492] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:07.178238 7102 tablet_bootstrap.cc:654] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.178268 7110 tablet_bootstrap.cc:654] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.179128 7107 ts_tablet_manager.cc:1398] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:07.179325 7103 tablet_bootstrap.cc:492] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:07.179421 7103 ts_tablet_manager.cc:1367] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:07.179862 7110 tablet_bootstrap.cc:492] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:07.179858 7103 raft_consensus.cc:348] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:07.179949 7110 ts_tablet_manager.cc:1367] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:07.179950 7103 raft_consensus.cc:374] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.179982 7103 raft_consensus.cc:729] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.180094 7103 consensus_queue.cc:260] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:07.180321 7103 ts_tablet_manager.cc:1398] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.180343 7110 raft_consensus.cc:348] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:07.180425 7110 raft_consensus.cc:374] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.180451 7102 tablet_bootstrap.cc:492] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:07.180461 7110 raft_consensus.cc:729] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.180523 7102 ts_tablet_manager.cc:1367] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:07.180577 7110 consensus_queue.cc:260] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:07.180837 7110 ts_tablet_manager.cc:1398] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.180902 7102 raft_consensus.cc:348] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:07.180987 7102 raft_consensus.cc:374] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.181012 7102 raft_consensus.cc:729] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.181155 7102 consensus_queue.cc:260] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:07.181381 7102 ts_tablet_manager.cc:1398] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.181852 7111 tablet_bootstrap.cc:492] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:07.182802 7111 tablet_bootstrap.cc:654] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.184242 7106 tablet_bootstrap.cc:492] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:07.184335 7106 ts_tablet_manager.cc:1367] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.000s
I20240501 19:56:07.184814 7106 raft_consensus.cc:348] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:07.184950 7106 raft_consensus.cc:374] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.184984 7106 raft_consensus.cc:729] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.185227 7106 consensus_queue.cc:260] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:07.185544 7106 ts_tablet_manager.cc:1398] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.187003 7111 tablet_bootstrap.cc:492] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:07.187088 7111 ts_tablet_manager.cc:1367] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:07.187505 7111 raft_consensus.cc:348] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:07.187592 7111 raft_consensus.cc:374] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.187626 7111 raft_consensus.cc:729] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.187752 7111 consensus_queue.cc:260] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:07.188037 7111 ts_tablet_manager.cc:1398] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.189085 6626 raft_consensus.cc:482] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:07.189162 6626 raft_consensus.cc:504] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:07.189484 6626 leader_election.cc:288] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:07.189831 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee048890b02446b1a7cdd28b3a54e223" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:07.189960 6260 raft_consensus.cc:2454] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:07.189949 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee048890b02446b1a7cdd28b3a54e223" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:07.190053 6523 raft_consensus.cc:2454] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:07.190264 5931 leader_election.cc:302] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:07.190727 6626 raft_consensus.cc:2781] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:07.190776 6626 raft_consensus.cc:482] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:07.190815 6626 raft_consensus.cc:3037] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.191434 6626 raft_consensus.cc:504] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:07.191740 6626 leader_election.cc:288] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:07.191998 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee048890b02446b1a7cdd28b3a54e223" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:07.192008 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee048890b02446b1a7cdd28b3a54e223" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:07.192095 6260 raft_consensus.cc:3037] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.192102 6523 raft_consensus.cc:3037] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.192819 6523 raft_consensus.cc:2454] T ee048890b02446b1a7cdd28b3a54e223 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:07.192924 6260 raft_consensus.cc:2454] T ee048890b02446b1a7cdd28b3a54e223 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:07.193127 5930 leader_election.cc:302] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:07.193315 6626 raft_consensus.cc:2781] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:07.193498 6626 raft_consensus.cc:686] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [term 1 LEADER]: Becoming Leader. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Running, Role: LEADER
I20240501 19:56:07.193619 6626 consensus_queue.cc:237] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:07.195192 5716 catalog_manager.cc:5484] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3489b69a0428491e957e46effb9b1146 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "3489b69a0428491e957e46effb9b1146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:07.321298 7011 raft_consensus.cc:482] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:07.321424 7011 raft_consensus.cc:504] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:07.321755 7011 leader_election.cc:288] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:07.322096 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d61f765fc274f2d841c0843ec8cc1fa" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:07.322178 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d61f765fc274f2d841c0843ec8cc1fa" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:07.322260 6523 raft_consensus.cc:2454] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:07.322307 5994 raft_consensus.cc:2454] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:07.322542 6195 leader_election.cc:302] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:07.322700 7011 raft_consensus.cc:2781] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:07.322757 7011 raft_consensus.cc:482] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:07.322814 7011 raft_consensus.cc:3037] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.325484 7011 raft_consensus.cc:504] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:07.325839 7011 leader_election.cc:288] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:07.326095 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d61f765fc274f2d841c0843ec8cc1fa" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:07.326206 6523 raft_consensus.cc:3037] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.326169 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d61f765fc274f2d841c0843ec8cc1fa" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:07.326287 5994 raft_consensus.cc:3037] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.326922 6523 raft_consensus.cc:2454] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:07.327153 5994 raft_consensus.cc:2454] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:07.327195 6195 leader_election.cc:302] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:07.327400 7011 raft_consensus.cc:2781] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:07.327507 7011 raft_consensus.cc:686] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [term 1 LEADER]: Becoming Leader. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Running, Role: LEADER
I20240501 19:56:07.327613 7011 consensus_queue.cc:237] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:07.329288 5716 catalog_manager.cc:5484] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1af421bf6a480587c69697af9f39dd (127.5.139.68). New cstate: current_term: 1 leader_uuid: "fb1af421bf6a480587c69697af9f39dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:07.489733 6608 raft_consensus.cc:482] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:07.489857 6608 raft_consensus.cc:504] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:07.490288 6608 leader_election.cc:288] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511), 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845)
I20240501 19:56:07.490626 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f41918d1c0094ec9aa4b6d1b8c57b9d2" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:07.490751 6392 raft_consensus.cc:2454] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a1d3564b3b40de9e4e4724bfa59d65 in term 0.
I20240501 19:56:07.490697 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f41918d1c0094ec9aa4b6d1b8c57b9d2" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:07.490929 5863 raft_consensus.cc:2454] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a1d3564b3b40de9e4e4724bfa59d65 in term 0.
I20240501 19:56:07.491101 6062 leader_election.cc:302] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:07.491258 6608 raft_consensus.cc:2781] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:07.491322 6608 raft_consensus.cc:482] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:07.491348 6608 raft_consensus.cc:3037] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.492421 6608 raft_consensus.cc:504] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:07.492750 6608 leader_election.cc:288] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 election: Requested vote from peers a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511), 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845)
I20240501 19:56:07.493005 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f41918d1c0094ec9aa4b6d1b8c57b9d2" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:07.493120 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f41918d1c0094ec9aa4b6d1b8c57b9d2" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44"
I20240501 19:56:07.493165 6392 raft_consensus.cc:3037] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.493247 5863 raft_consensus.cc:3037] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.493952 6392 raft_consensus.cc:2454] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a1d3564b3b40de9e4e4724bfa59d65 in term 1.
I20240501 19:56:07.493974 5863 raft_consensus.cc:2454] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a1d3564b3b40de9e4e4724bfa59d65 in term 1.
I20240501 19:56:07.494220 6062 leader_election.cc:302] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:07.494375 6608 raft_consensus.cc:2781] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:07.494562 6608 raft_consensus.cc:686] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 LEADER]: Becoming Leader. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Running, Role: LEADER
I20240501 19:56:07.494891 6608 consensus_queue.cc:237] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:07.497463 5716 catalog_manager.cc:5484] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67). New cstate: current_term: 1 leader_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:07.639199 6626 consensus_queue.cc:1035] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:07.639958 7122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:07.640204 7122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 19:56:07.643121 6626 consensus_queue.cc:1035] T ee048890b02446b1a7cdd28b3a54e223 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:07.644872 7122 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 19:56:07.755005 5994 raft_consensus.cc:1261] T 8d61f765fc274f2d841c0843ec8cc1fa P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.755635 7011 consensus_queue.cc:1035] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.759502 5863 raft_consensus.cc:1261] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Refusing update from remote peer d6a1d3564b3b40de9e4e4724bfa59d65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.759526 6392 raft_consensus.cc:1261] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer d6a1d3564b3b40de9e4e4724bfa59d65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.759992 6608 consensus_queue.cc:1035] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.760393 6608 consensus_queue.cc:1035] T f41918d1c0094ec9aa4b6d1b8c57b9d2 P d6a1d3564b3b40de9e4e4724bfa59d65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:07.764278 6523 raft_consensus.cc:1261] T 8d61f765fc274f2d841c0843ec8cc1fa P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:07.770220 7011 consensus_queue.cc:1035] T 8d61f765fc274f2d841c0843ec8cc1fa P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 174.21 ms
time per row: 0.0871048 ms
I20240501 19:56:07.862576 5719 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:07.867820 6106 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb5dbfc92754524910519306c02b132 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c4db1ae8abfc4ef491d29978c2fa7fcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.867831 5843 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb5dbfc92754524910519306c02b132 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c4db1ae8abfc4ef491d29978c2fa7fcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.868156 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb5dbfc92754524910519306c02b132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.868163 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb5dbfc92754524910519306c02b132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.868445 6239 tablet_service.cc:1467] Processing CreateTablet for tablet 631040a5e0114513b578a6794a69bab5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c4db1ae8abfc4ef491d29978c2fa7fcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.868693 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631040a5e0114513b578a6794a69bab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.870041 7174 tablet_bootstrap.cc:492] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:07.870448 5842 tablet_service.cc:1467] Processing CreateTablet for tablet b3c88941cbe044e8954d560f52b0695d (DEFAULT_TABLE table=rebalance_test_table_8 [id=c4db1ae8abfc4ef491d29978c2fa7fcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.870676 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c88941cbe044e8954d560f52b0695d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.871253 7174 tablet_bootstrap.cc:654] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.871363 6504 tablet_service.cc:1467] Processing CreateTablet for tablet 631040a5e0114513b578a6794a69bab5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c4db1ae8abfc4ef491d29978c2fa7fcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.871649 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631040a5e0114513b578a6794a69bab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.871618 6371 tablet_service.cc:1467] Processing CreateTablet for tablet b3c88941cbe044e8954d560f52b0695d (DEFAULT_TABLE table=rebalance_test_table_8 [id=c4db1ae8abfc4ef491d29978c2fa7fcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.871896 6371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c88941cbe044e8954d560f52b0695d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.872717 6106 tablet_service.cc:1467] Processing CreateTablet for tablet b3c88941cbe044e8954d560f52b0695d (DEFAULT_TABLE table=rebalance_test_table_8 [id=c4db1ae8abfc4ef491d29978c2fa7fcd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.872958 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3c88941cbe044e8954d560f52b0695d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.873399 7175 tablet_bootstrap.cc:492] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:07.874202 6372 tablet_service.cc:1467] Processing CreateTablet for tablet 9fb5dbfc92754524910519306c02b132 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c4db1ae8abfc4ef491d29978c2fa7fcd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.874279 7175 tablet_bootstrap.cc:654] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.874455 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fb5dbfc92754524910519306c02b132. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.874626 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 631040a5e0114513b578a6794a69bab5 (DEFAULT_TABLE table=rebalance_test_table_8 [id=c4db1ae8abfc4ef491d29978c2fa7fcd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:07.874948 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 631040a5e0114513b578a6794a69bab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:07.875080 7178 tablet_bootstrap.cc:492] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:07.876272 7178 tablet_bootstrap.cc:654] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.876910 7179 tablet_bootstrap.cc:492] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:07.876986 7180 tablet_bootstrap.cc:492] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:07.877714 7179 tablet_bootstrap.cc:654] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.877882 7180 tablet_bootstrap.cc:654] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.878741 7174 tablet_bootstrap.cc:492] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:07.878852 7179 tablet_bootstrap.cc:492] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:07.878887 7174 ts_tablet_manager.cc:1367] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:07.878954 7176 tablet_bootstrap.cc:492] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:07.879004 7179 ts_tablet_manager.cc:1367] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:07.879279 7180 tablet_bootstrap.cc:492] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:07.879359 7180 ts_tablet_manager.cc:1367] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:07.879375 7179 raft_consensus.cc:348] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.879366 7174 raft_consensus.cc:348] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.879472 7179 raft_consensus.cc:374] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.879477 7174 raft_consensus.cc:374] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.879508 7174 raft_consensus.cc:729] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.879508 7179 raft_consensus.cc:729] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.879622 7174 consensus_queue.cc:260] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.879622 7179 consensus_queue.cc:260] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.879770 7180 raft_consensus.cc:348] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:07.879881 7180 raft_consensus.cc:374] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.879897 7174 ts_tablet_manager.cc:1398] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.879912 7180 raft_consensus.cc:729] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.880038 7174 tablet_bootstrap.cc:492] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:07.880028 7180 consensus_queue.cc:260] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:07.880173 7178 tablet_bootstrap.cc:492] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:07.880306 7180 ts_tablet_manager.cc:1398] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.880246 7178 ts_tablet_manager.cc:1367] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:07.880461 7179 ts_tablet_manager.cc:1398] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.880611 7179 tablet_bootstrap.cc:492] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:07.880893 7174 tablet_bootstrap.cc:654] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.881443 7179 tablet_bootstrap.cc:654] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.881697 7178 raft_consensus.cc:348] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:07.881820 7178 raft_consensus.cc:374] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.881855 7178 raft_consensus.cc:729] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.882324 7175 tablet_bootstrap.cc:492] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:07.882347 7176 tablet_bootstrap.cc:654] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.882473 7175 ts_tablet_manager.cc:1367] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 19:56:07.882998 7175 raft_consensus.cc:348] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.883167 7175 raft_consensus.cc:374] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.883253 7175 raft_consensus.cc:729] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.883450 7175 consensus_queue.cc:260] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.884071 7174 tablet_bootstrap.cc:492] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:07.884163 7174 ts_tablet_manager.cc:1367] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:07.884364 7175 ts_tablet_manager.cc:1398] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:07.884537 7174 raft_consensus.cc:348] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.884640 7174 raft_consensus.cc:374] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.884683 7174 raft_consensus.cc:729] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.884809 7174 consensus_queue.cc:260] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.884538 7178 consensus_queue.cc:260] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:07.885103 7174 ts_tablet_manager.cc:1398] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:07.885452 7178 ts_tablet_manager.cc:1398] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:07.885962 7176 tablet_bootstrap.cc:492] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:07.886057 7176 ts_tablet_manager.cc:1367] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:07.886060 7175 tablet_bootstrap.cc:492] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:07.886495 7176 raft_consensus.cc:348] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:07.886626 7176 raft_consensus.cc:374] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.886667 7176 raft_consensus.cc:729] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.886821 7176 consensus_queue.cc:260] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:07.887105 7176 ts_tablet_manager.cc:1398] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.887408 7175 tablet_bootstrap.cc:654] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:07.888178 7179 tablet_bootstrap.cc:492] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:07.888270 7179 ts_tablet_manager.cc:1367] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.001s
I20240501 19:56:07.888675 7179 raft_consensus.cc:348] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.888800 7179 raft_consensus.cc:374] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.888836 7179 raft_consensus.cc:729] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.888948 7179 consensus_queue.cc:260] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.889261 7179 ts_tablet_manager.cc:1398] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:07.901401 7175 tablet_bootstrap.cc:492] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:07.901517 7175 ts_tablet_manager.cc:1367] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:56:07.902015 7175 raft_consensus.cc:348] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.902114 7175 raft_consensus.cc:374] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:07.902135 7175 raft_consensus.cc:729] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:07.902251 7175 consensus_queue.cc:260] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.902499 7175 ts_tablet_manager.cc:1398] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:07.934320 6622 raft_consensus.cc:482] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:07.934427 6622 raft_consensus.cc:504] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.934727 6622 leader_election.cc:288] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:07.935086 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5dbfc92754524910519306c02b132" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:07.935125 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5dbfc92754524910519306c02b132" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:07.935225 6126 raft_consensus.cc:2454] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:07.935261 6392 raft_consensus.cc:2454] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:07.935519 5798 leader_election.cc:302] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:07.935700 6622 raft_consensus.cc:2781] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:07.935770 6622 raft_consensus.cc:482] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:07.935815 6622 raft_consensus.cc:3037] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.936599 6622 raft_consensus.cc:504] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.936949 6622 leader_election.cc:288] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Requested vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:07.937270 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5dbfc92754524910519306c02b132" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:07.937328 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fb5dbfc92754524910519306c02b132" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:07.937388 6126 raft_consensus.cc:3037] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.937453 6392 raft_consensus.cc:3037] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.938165 6392 raft_consensus.cc:2454] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:07.938359 6126 raft_consensus.cc:2454] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:07.938421 5798 leader_election.cc:302] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:07.938578 6622 raft_consensus.cc:2781] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:07.938779 6622 raft_consensus.cc:686] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [term 1 LEADER]: Becoming Leader. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Running, Role: LEADER
I20240501 19:56:07.938941 6622 consensus_queue.cc:237] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:07.940596 5713 catalog_manager.cc:5484] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4629e756401746a8bf4652f995bddb44 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "4629e756401746a8bf4652f995bddb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:07.991022 7123 raft_consensus.cc:482] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:07.991155 7123 raft_consensus.cc:504] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:07.991575 7123 leader_election.cc:288] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:07.991943 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631040a5e0114513b578a6794a69bab5" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:07.991950 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631040a5e0114513b578a6794a69bab5" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:07.992079 6260 raft_consensus.cc:2454] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:07.992085 5994 raft_consensus.cc:2454] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:07.992336 6460 leader_election.cc:302] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:07.992492 7123 raft_consensus.cc:2781] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:07.992553 7123 raft_consensus.cc:482] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:07.992599 7123 raft_consensus.cc:3037] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.993479 7123 raft_consensus.cc:504] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:07.993826 7123 leader_election.cc:288] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Requested vote from peers 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:07.994180 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631040a5e0114513b578a6794a69bab5" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:07.994261 6260 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "631040a5e0114513b578a6794a69bab5" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:07.994307 5994 raft_consensus.cc:3037] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.994378 6260 raft_consensus.cc:3037] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:07.995077 6260 raft_consensus.cc:2454] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:07.995185 5994 raft_consensus.cc:2454] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:07.995401 6460 leader_election.cc:302] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:07.995759 7123 raft_consensus.cc:2781] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:07.995863 7123 raft_consensus.cc:686] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 LEADER]: Becoming Leader. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Running, Role: LEADER
I20240501 19:56:07.996001 7123 consensus_queue.cc:237] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:07.997881 5713 catalog_manager.cc:5484] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70). New cstate: current_term: 1 leader_uuid: "60b2c1ef2e86405bba90eaa1db142e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.006752 6622 raft_consensus.cc:482] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:08.006853 6622 raft_consensus.cc:504] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:08.007151 6622 leader_election.cc:288] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:08.007555 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c88941cbe044e8954d560f52b0695d" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:08.007606 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c88941cbe044e8954d560f52b0695d" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:08.007720 6126 raft_consensus.cc:2454] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:08.007725 6392 raft_consensus.cc:2454] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:08.007978 5798 leader_election.cc:302] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:08.008143 6622 raft_consensus.cc:2781] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:08.008203 6622 raft_consensus.cc:482] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:08.008247 6622 raft_consensus.cc:3037] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.009021 6622 raft_consensus.cc:504] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:08.009349 6622 leader_election.cc:288] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Requested vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:08.009680 6126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c88941cbe044e8954d560f52b0695d" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:08.009717 6392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3c88941cbe044e8954d560f52b0695d" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:08.009821 6126 raft_consensus.cc:3037] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.009821 6392 raft_consensus.cc:3037] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.010702 6392 raft_consensus.cc:2454] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:08.010701 6126 raft_consensus.cc:2454] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:08.011003 5798 leader_election.cc:302] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:08.011162 6622 raft_consensus.cc:2781] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:08.011250 6622 raft_consensus.cc:686] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [term 1 LEADER]: Becoming Leader. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Running, Role: LEADER
I20240501 19:56:08.011389 6622 consensus_queue.cc:237] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:08.013170 5713 catalog_manager.cc:5484] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4629e756401746a8bf4652f995bddb44 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "4629e756401746a8bf4652f995bddb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:08.138231 7194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:08.138458 7194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:08.141412 7194 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 19:56:08.228870 6392 raft_consensus.cc:1261] T 9fb5dbfc92754524910519306c02b132 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.228930 6126 raft_consensus.cc:1261] T 9fb5dbfc92754524910519306c02b132 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.229908 5994 raft_consensus.cc:1261] T 631040a5e0114513b578a6794a69bab5 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.230027 6260 raft_consensus.cc:1261] T 631040a5e0114513b578a6794a69bab5 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.230408 6600 consensus_queue.cc:1035] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.230547 7123 consensus_queue.cc:1035] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.230829 7190 consensus_queue.cc:1035] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.231179 7190 consensus_queue.cc:1035] T 9fb5dbfc92754524910519306c02b132 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.231912 6392 raft_consensus.cc:1261] T b3c88941cbe044e8954d560f52b0695d P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.233681 6622 consensus_queue.cc:1035] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.239096 6127 raft_consensus.cc:1261] T b3c88941cbe044e8954d560f52b0695d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.239639 7190 consensus_queue.cc:1035] T b3c88941cbe044e8954d560f52b0695d P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.240449 7158 mvcc.cc:205] Tried to move back new op lower bound from 7022974436258054144 to 7022974435079114752. Current Snapshot: MvccSnapshot[applied={T|T < 7022974436258054144}]
I20240501 19:56:08.252336 7160 mvcc.cc:205] Tried to move back new op lower bound from 7022974436258054144 to 7022974435079114752. Current Snapshot: MvccSnapshot[applied={T|T < 7022974436258054144}]
I20240501 19:56:08.253930 7131 mvcc.cc:205] Tried to move back new op lower bound from 7022974436264923136 to 7022974435312939008. Current Snapshot: MvccSnapshot[applied={T|T < 7022974436264923136}]
I20240501 19:56:08.261197 7156 mvcc.cc:205] Tried to move back new op lower bound from 7022974436258054144 to 7022974435079114752. Current Snapshot: MvccSnapshot[applied={T|T < 7022974436258054144}]
INSERT report
rows total: 2000
time total: 138.662 ms
time per row: 0.0693309 ms
I20240501 19:56:08.320603 5715 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:08.325969 6504 tablet_service.cc:1467] Processing CreateTablet for tablet 195246eb76cf4aa294aad0b658588a89 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1cf49c10c6f84ec2ab3e953f905785a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.326201 6503 tablet_service.cc:1467] Processing CreateTablet for tablet ecdd31a4e11847e1b747d5c4f6fc409f (DEFAULT_TABLE table=rebalance_test_table_9 [id=1cf49c10c6f84ec2ab3e953f905785a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.326333 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195246eb76cf4aa294aad0b658588a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.326460 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecdd31a4e11847e1b747d5c4f6fc409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.326483 6240 tablet_service.cc:1467] Processing CreateTablet for tablet ecdd31a4e11847e1b747d5c4f6fc409f (DEFAULT_TABLE table=rebalance_test_table_9 [id=1cf49c10c6f84ec2ab3e953f905785a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.326614 6239 tablet_service.cc:1467] Processing CreateTablet for tablet 195246eb76cf4aa294aad0b658588a89 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1cf49c10c6f84ec2ab3e953f905785a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.326743 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecdd31a4e11847e1b747d5c4f6fc409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.326859 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195246eb76cf4aa294aad0b658588a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.328209 7178 tablet_bootstrap.cc:492] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:08.328706 5974 tablet_service.cc:1467] Processing CreateTablet for tablet 195246eb76cf4aa294aad0b658588a89 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1cf49c10c6f84ec2ab3e953f905785a3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.329007 5974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 195246eb76cf4aa294aad0b658588a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.329368 7178 tablet_bootstrap.cc:654] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.329522 5975 tablet_service.cc:1467] Processing CreateTablet for tablet ecdd31a4e11847e1b747d5c4f6fc409f (DEFAULT_TABLE table=rebalance_test_table_9 [id=1cf49c10c6f84ec2ab3e953f905785a3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.329730 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ecdd31a4e11847e1b747d5c4f6fc409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.329711 6372 tablet_service.cc:1467] Processing CreateTablet for tablet 7112210d92834e2c96c2f6ffe614ec98 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1cf49c10c6f84ec2ab3e953f905785a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.329744 6106 tablet_service.cc:1467] Processing CreateTablet for tablet 7112210d92834e2c96c2f6ffe614ec98 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1cf49c10c6f84ec2ab3e953f905785a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.329988 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7112210d92834e2c96c2f6ffe614ec98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.330026 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7112210d92834e2c96c2f6ffe614ec98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.330049 7176 tablet_bootstrap.cc:492] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:08.330253 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 7112210d92834e2c96c2f6ffe614ec98 (DEFAULT_TABLE table=rebalance_test_table_9 [id=1cf49c10c6f84ec2ab3e953f905785a3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.330515 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7112210d92834e2c96c2f6ffe614ec98. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.330852 7176 tablet_bootstrap.cc:654] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.332181 7179 tablet_bootstrap.cc:492] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:08.332386 7174 tablet_bootstrap.cc:492] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:08.332619 7175 tablet_bootstrap.cc:492] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:08.332898 7179 tablet_bootstrap.cc:654] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.333233 7174 tablet_bootstrap.cc:654] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.333530 7175 tablet_bootstrap.cc:654] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.334607 7179 tablet_bootstrap.cc:492] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:08.334676 7179 ts_tablet_manager.cc:1367] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:08.335134 7175 tablet_bootstrap.cc:492] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:08.335067 7179 raft_consensus.cc:348] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.335176 7179 raft_consensus.cc:374] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.335215 7179 raft_consensus.cc:729] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.335216 7175 ts_tablet_manager.cc:1367] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:08.335343 7179 consensus_queue.cc:260] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.335572 7179 ts_tablet_manager.cc:1398] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:08.335680 7175 raft_consensus.cc:348] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.335808 7175 raft_consensus.cc:374] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.335853 7175 raft_consensus.cc:729] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.335996 7175 consensus_queue.cc:260] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.336261 7175 ts_tablet_manager.cc:1398] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:08.336390 7178 tablet_bootstrap.cc:492] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:08.336473 7178 ts_tablet_manager.cc:1367] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:08.336871 7178 raft_consensus.cc:348] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:08.336987 7178 raft_consensus.cc:374] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.337013 7178 raft_consensus.cc:729] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.337267 7174 tablet_bootstrap.cc:492] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:08.337282 7178 consensus_queue.cc:260] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:08.337353 7174 ts_tablet_manager.cc:1367] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:08.337553 7178 ts_tablet_manager.cc:1398] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:08.337720 7178 tablet_bootstrap.cc:492] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:08.337771 7174 raft_consensus.cc:348] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.337870 7174 raft_consensus.cc:374] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.337908 7174 raft_consensus.cc:729] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.338028 7174 consensus_queue.cc:260] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.338275 7174 ts_tablet_manager.cc:1398] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:08.338548 7178 tablet_bootstrap.cc:654] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.340934 7178 tablet_bootstrap.cc:492] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:08.341446 7178 ts_tablet_manager.cc:1367] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:08.342049 7178 raft_consensus.cc:348] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:08.342207 7178 raft_consensus.cc:374] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.342281 7178 raft_consensus.cc:729] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.342453 7178 consensus_queue.cc:260] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:08.342697 7178 ts_tablet_manager.cc:1398] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:08.345458 7176 tablet_bootstrap.cc:492] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:08.345549 7176 ts_tablet_manager.cc:1367] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.000s
I20240501 19:56:08.346014 7176 raft_consensus.cc:348] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:08.346114 7176 raft_consensus.cc:374] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.346148 7176 raft_consensus.cc:729] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.346267 7176 consensus_queue.cc:260] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:08.346506 7176 ts_tablet_manager.cc:1398] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:08.347295 7180 tablet_bootstrap.cc:492] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:08.348170 7180 tablet_bootstrap.cc:654] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.349001 7176 tablet_bootstrap.cc:492] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:08.349665 7180 tablet_bootstrap.cc:492] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:08.349737 7180 ts_tablet_manager.cc:1367] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:08.350016 7180 raft_consensus.cc:348] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:08.350085 7180 raft_consensus.cc:374] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.350103 7180 raft_consensus.cc:729] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.350210 7176 tablet_bootstrap.cc:654] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.350204 7180 consensus_queue.cc:260] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:08.350443 7180 ts_tablet_manager.cc:1398] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:08.350634 7180 tablet_bootstrap.cc:492] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:08.351325 7180 tablet_bootstrap.cc:654] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.352653 7180 tablet_bootstrap.cc:492] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:08.352742 7180 ts_tablet_manager.cc:1367] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:08.353183 7180 raft_consensus.cc:348] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:08.353336 7180 raft_consensus.cc:374] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.353416 7180 raft_consensus.cc:729] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.353611 7180 consensus_queue.cc:260] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:08.353963 7180 ts_tablet_manager.cc:1398] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:08.357587 7176 tablet_bootstrap.cc:492] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:08.357669 7176 ts_tablet_manager.cc:1367] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240501 19:56:08.358060 7176 raft_consensus.cc:348] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:08.358158 7176 raft_consensus.cc:374] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.358196 7176 raft_consensus.cc:729] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.358311 7176 consensus_queue.cc:260] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:08.358639 7176 ts_tablet_manager.cc:1398] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:08.362437 6753 raft_consensus.cc:482] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:08.362507 6753 raft_consensus.cc:504] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:08.362768 6753 leader_election.cc:288] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:08.363097 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecdd31a4e11847e1b747d5c4f6fc409f" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:08.363140 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecdd31a4e11847e1b747d5c4f6fc409f" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:08.363232 6523 raft_consensus.cc:2454] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:08.363231 5994 raft_consensus.cc:2454] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:08.363478 6195 leader_election.cc:302] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:08.363623 6753 raft_consensus.cc:2781] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:08.363678 6753 raft_consensus.cc:482] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:08.363745 6753 raft_consensus.cc:3037] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.364428 6753 raft_consensus.cc:504] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:08.364706 6753 leader_election.cc:288] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:08.364972 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecdd31a4e11847e1b747d5c4f6fc409f" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:08.365025 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ecdd31a4e11847e1b747d5c4f6fc409f" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:08.365103 6523 raft_consensus.cc:3037] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.365136 5994 raft_consensus.cc:3037] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.365932 6523 raft_consensus.cc:2454] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:08.365932 5994 raft_consensus.cc:2454] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:08.366197 6195 leader_election.cc:302] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:08.366345 6753 raft_consensus.cc:2781] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:08.366434 6753 raft_consensus.cc:686] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [term 1 LEADER]: Becoming Leader. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Running, Role: LEADER
I20240501 19:56:08.366567 6753 consensus_queue.cc:237] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:08.368093 5715 catalog_manager.cc:5484] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1af421bf6a480587c69697af9f39dd (127.5.139.68). New cstate: current_term: 1 leader_uuid: "fb1af421bf6a480587c69697af9f39dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.448980 7231 raft_consensus.cc:482] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:08.449151 7231 raft_consensus.cc:504] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.449700 7231 leader_election.cc:288] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:08.450093 6391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7112210d92834e2c96c2f6ffe614ec98" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:08.450114 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7112210d92834e2c96c2f6ffe614ec98" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:08.450227 6391 raft_consensus.cc:2454] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:08.450248 6127 raft_consensus.cc:2454] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:08.450536 5798 leader_election.cc:302] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:08.450726 7231 raft_consensus.cc:2781] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:08.450785 7231 raft_consensus.cc:482] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:08.450812 7231 raft_consensus.cc:3037] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.451643 7231 raft_consensus.cc:504] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.451933 7231 leader_election.cc:288] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Requested vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:08.452279 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7112210d92834e2c96c2f6ffe614ec98" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:08.452325 6391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7112210d92834e2c96c2f6ffe614ec98" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:08.452440 6391 raft_consensus.cc:3037] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.452440 6127 raft_consensus.cc:3037] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.453433 6391 raft_consensus.cc:2454] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:08.453433 6127 raft_consensus.cc:2454] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:08.453753 5798 leader_election.cc:302] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:08.453928 7231 raft_consensus.cc:2781] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:08.454010 7231 raft_consensus.cc:686] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [term 1 LEADER]: Becoming Leader. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Running, Role: LEADER
I20240501 19:56:08.454146 7231 consensus_queue.cc:237] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.455830 5715 catalog_manager.cc:5484] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4629e756401746a8bf4652f995bddb44 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "4629e756401746a8bf4652f995bddb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.501392 6753 raft_consensus.cc:482] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:08.501528 6753 raft_consensus.cc:504] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:08.501972 6753 leader_election.cc:288] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:08.502446 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195246eb76cf4aa294aad0b658588a89" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:08.502449 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195246eb76cf4aa294aad0b658588a89" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:08.502589 6523 raft_consensus.cc:2454] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:08.502590 5994 raft_consensus.cc:2454] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 0.
I20240501 19:56:08.502884 6195 leader_election.cc:302] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:08.503093 6753 raft_consensus.cc:2781] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:08.503160 6753 raft_consensus.cc:482] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:08.503190 6753 raft_consensus.cc:3037] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.504004 6753 raft_consensus.cc:504] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:08.504320 6753 leader_election.cc:288] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:08.504652 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195246eb76cf4aa294aad0b658588a89" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:08.504740 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "195246eb76cf4aa294aad0b658588a89" candidate_uuid: "fb1af421bf6a480587c69697af9f39dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:08.504782 6523 raft_consensus.cc:3037] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.504855 5994 raft_consensus.cc:3037] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.505581 6523 raft_consensus.cc:2454] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:08.505600 5994 raft_consensus.cc:2454] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb1af421bf6a480587c69697af9f39dd in term 1.
I20240501 19:56:08.505861 6195 leader_election.cc:302] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:08.506083 6753 raft_consensus.cc:2781] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:08.506254 6753 raft_consensus.cc:686] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [term 1 LEADER]: Becoming Leader. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Running, Role: LEADER
I20240501 19:56:08.506436 6753 consensus_queue.cc:237] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:08.508169 5715 catalog_manager.cc:5484] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd reported cstate change: term changed from 0 to 1, leader changed from <none> to fb1af421bf6a480587c69697af9f39dd (127.5.139.68). New cstate: current_term: 1 leader_uuid: "fb1af421bf6a480587c69697af9f39dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:08.636684 7245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:08.636920 7245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:08.639956 7245 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 19:56:08.717972 5994 raft_consensus.cc:1261] T ecdd31a4e11847e1b747d5c4f6fc409f P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.718564 6753 consensus_queue.cc:1035] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.718639 6391 raft_consensus.cc:1261] T 7112210d92834e2c96c2f6ffe614ec98 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.718966 6127 raft_consensus.cc:1261] T 7112210d92834e2c96c2f6ffe614ec98 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.719396 6622 consensus_queue.cc:1035] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.720242 6622 consensus_queue.cc:1035] T 7112210d92834e2c96c2f6ffe614ec98 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.725426 6523 raft_consensus.cc:1261] T 195246eb76cf4aa294aad0b658588a89 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.726289 7170 consensus_queue.cc:1035] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.728164 6524 raft_consensus.cc:1261] T ecdd31a4e11847e1b747d5c4f6fc409f P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.729279 7170 consensus_queue.cc:1035] T ecdd31a4e11847e1b747d5c4f6fc409f P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.729676 5995 raft_consensus.cc:1261] T 195246eb76cf4aa294aad0b658588a89 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer fb1af421bf6a480587c69697af9f39dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:08.735276 7170 consensus_queue.cc:1035] T 195246eb76cf4aa294aad0b658588a89 P fb1af421bf6a480587c69697af9f39dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:08.738480 7129 mvcc.cc:205] Tried to move back new op lower bound from 7022974438263201792 to 7022974436830904320. Current Snapshot: MvccSnapshot[applied={T|T < 7022974438263201792}]
I20240501 19:56:08.769353 7167 mvcc.cc:205] Tried to move back new op lower bound from 7022974438283804672 to 7022974437403611136. Current Snapshot: MvccSnapshot[applied={T|T < 7022974438283804672}]
INSERT report
rows total: 2000
time total: 105.965 ms
time per row: 0.0529827 ms
I20240501 19:56:08.786564 5719 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:08.792104 6372 tablet_service.cc:1467] Processing CreateTablet for tablet 1db84f3b4048482f9949a97861defc33 (DEFAULT_TABLE table=rebalance_test_table_10 [id=819bdb11a1df48fd8b6d7d0108d7ccd2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.792094 5842 tablet_service.cc:1467] Processing CreateTablet for tablet 1db84f3b4048482f9949a97861defc33 (DEFAULT_TABLE table=rebalance_test_table_10 [id=819bdb11a1df48fd8b6d7d0108d7ccd2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.792413 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db84f3b4048482f9949a97861defc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.792413 5842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db84f3b4048482f9949a97861defc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.792845 6239 tablet_service.cc:1467] Processing CreateTablet for tablet 935a6d3444ea4cdbb6ab26271ac3bcf2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=819bdb11a1df48fd8b6d7d0108d7ccd2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.793097 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935a6d3444ea4cdbb6ab26271ac3bcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.794111 7175 tablet_bootstrap.cc:492] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:08.794896 5975 tablet_service.cc:1467] Processing CreateTablet for tablet 935a6d3444ea4cdbb6ab26271ac3bcf2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=819bdb11a1df48fd8b6d7d0108d7ccd2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.794975 7175 tablet_bootstrap.cc:654] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.795166 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935a6d3444ea4cdbb6ab26271ac3bcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.795250 5843 tablet_service.cc:1467] Processing CreateTablet for tablet e7f65dcb7f9e43d6959a9cf91f6e51c7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=819bdb11a1df48fd8b6d7d0108d7ccd2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.795485 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f65dcb7f9e43d6959a9cf91f6e51c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.795775 6106 tablet_service.cc:1467] Processing CreateTablet for tablet 1db84f3b4048482f9949a97861defc33 (DEFAULT_TABLE table=rebalance_test_table_10 [id=819bdb11a1df48fd8b6d7d0108d7ccd2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.795817 6107 tablet_service.cc:1467] Processing CreateTablet for tablet e7f65dcb7f9e43d6959a9cf91f6e51c7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=819bdb11a1df48fd8b6d7d0108d7ccd2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.796007 6106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1db84f3b4048482f9949a97861defc33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.796175 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f65dcb7f9e43d6959a9cf91f6e51c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.797346 7179 tablet_bootstrap.cc:492] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:08.797451 6503 tablet_service.cc:1467] Processing CreateTablet for tablet 935a6d3444ea4cdbb6ab26271ac3bcf2 (DEFAULT_TABLE table=rebalance_test_table_10 [id=819bdb11a1df48fd8b6d7d0108d7ccd2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.797696 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 935a6d3444ea4cdbb6ab26271ac3bcf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.798295 7179 tablet_bootstrap.cc:654] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.798300 7174 tablet_bootstrap.cc:492] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:08.798872 7176 tablet_bootstrap.cc:492] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:08.799181 7174 tablet_bootstrap.cc:654] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.799516 7176 tablet_bootstrap.cc:654] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.799765 7178 tablet_bootstrap.cc:492] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:08.799821 7180 tablet_bootstrap.cc:492] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:08.800575 7179 tablet_bootstrap.cc:492] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:08.800606 7178 tablet_bootstrap.cc:654] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.800673 7179 ts_tablet_manager.cc:1367] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:08.800737 7180 tablet_bootstrap.cc:654] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.801151 7174 tablet_bootstrap.cc:492] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:08.801249 7174 ts_tablet_manager.cc:1367] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:08.801630 7179 raft_consensus.cc:348] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:08.801750 7179 raft_consensus.cc:374] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.801784 7179 raft_consensus.cc:729] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.801761 7174 raft_consensus.cc:348] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:08.801836 6372 tablet_service.cc:1467] Processing CreateTablet for tablet e7f65dcb7f9e43d6959a9cf91f6e51c7 (DEFAULT_TABLE table=rebalance_test_table_10 [id=819bdb11a1df48fd8b6d7d0108d7ccd2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:08.801889 7174 raft_consensus.cc:374] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.801926 7174 raft_consensus.cc:729] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.801898 7179 consensus_queue.cc:260] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:08.802034 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7f65dcb7f9e43d6959a9cf91f6e51c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:08.802047 7174 consensus_queue.cc:260] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:08.802161 7179 ts_tablet_manager.cc:1398] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:08.802314 7174 ts_tablet_manager.cc:1398] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:08.802481 7174 tablet_bootstrap.cc:492] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:08.802798 7176 tablet_bootstrap.cc:492] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:08.802830 7178 tablet_bootstrap.cc:492] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:08.802883 7176 ts_tablet_manager.cc:1367] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:08.802904 7178 ts_tablet_manager.cc:1367] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:08.803297 7174 tablet_bootstrap.cc:654] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.803323 7176 raft_consensus.cc:348] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:08.803337 7178 raft_consensus.cc:348] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:08.803417 7176 raft_consensus.cc:374] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.803426 7178 raft_consensus.cc:374] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.803454 7178 raft_consensus.cc:729] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.803455 7176 raft_consensus.cc:729] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.803570 7178 consensus_queue.cc:260] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:08.803584 7176 consensus_queue.cc:260] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:08.803874 7178 ts_tablet_manager.cc:1398] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:08.803881 7176 ts_tablet_manager.cc:1398] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:08.804641 7179 tablet_bootstrap.cc:492] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:08.805524 7179 tablet_bootstrap.cc:654] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.806689 7180 tablet_bootstrap.cc:492] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:08.806783 7180 ts_tablet_manager.cc:1367] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:08.807242 7180 raft_consensus.cc:348] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:08.807343 7180 raft_consensus.cc:374] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.807374 7180 raft_consensus.cc:729] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.807492 7180 consensus_queue.cc:260] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:08.807806 7180 ts_tablet_manager.cc:1398] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:08.807848 7179 tablet_bootstrap.cc:492] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:08.807935 7179 ts_tablet_manager.cc:1367] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:08.808336 7179 raft_consensus.cc:348] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.808437 7179 raft_consensus.cc:374] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.808477 7179 raft_consensus.cc:729] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.808599 7179 consensus_queue.cc:260] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.808832 7175 tablet_bootstrap.cc:492] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:08.808902 7179 ts_tablet_manager.cc:1398] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:08.808930 7175 ts_tablet_manager.cc:1367] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:56:08.809578 7175 raft_consensus.cc:348] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:08.809680 7175 raft_consensus.cc:374] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.809718 7175 raft_consensus.cc:729] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.809828 7175 consensus_queue.cc:260] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:08.810111 7175 ts_tablet_manager.cc:1398] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:08.810830 7175 tablet_bootstrap.cc:492] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:08.811615 7175 tablet_bootstrap.cc:654] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:08.813164 7174 tablet_bootstrap.cc:492] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:08.813252 7174 ts_tablet_manager.cc:1367] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:08.813584 7174 raft_consensus.cc:348] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.813711 7174 raft_consensus.cc:374] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.813755 7174 raft_consensus.cc:729] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.813868 7174 consensus_queue.cc:260] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.814106 7174 ts_tablet_manager.cc:1398] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:08.814524 7175 tablet_bootstrap.cc:492] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:08.814610 7175 ts_tablet_manager.cc:1367] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240501 19:56:08.814960 7175 raft_consensus.cc:348] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.815034 7175 raft_consensus.cc:374] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:08.815054 7175 raft_consensus.cc:729] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:08.815147 7175 consensus_queue.cc:260] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:08.815408 7175 ts_tablet_manager.cc:1398] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:08.830132 7120 raft_consensus.cc:482] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:08.830224 7120 raft_consensus.cc:504] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:08.830569 7120 leader_election.cc:288] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:08.830817 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db84f3b4048482f9949a97861defc33" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:08.830853 6391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db84f3b4048482f9949a97861defc33" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:08.830935 5863 raft_consensus.cc:2454] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a1d3564b3b40de9e4e4724bfa59d65 in term 0.
I20240501 19:56:08.830962 6391 raft_consensus.cc:2454] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6a1d3564b3b40de9e4e4724bfa59d65 in term 0.
I20240501 19:56:08.831215 6063 leader_election.cc:302] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:08.831430 7120 raft_consensus.cc:2781] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:08.831478 7120 raft_consensus.cc:482] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:08.831506 7120 raft_consensus.cc:3037] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.832341 7120 raft_consensus.cc:504] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:08.832636 7120 leader_election.cc:288] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 election: Requested vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:08.832890 6391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db84f3b4048482f9949a97861defc33" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:08.832885 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1db84f3b4048482f9949a97861defc33" candidate_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44"
I20240501 19:56:08.832967 6391 raft_consensus.cc:3037] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.832973 5863 raft_consensus.cc:3037] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.833724 6391 raft_consensus.cc:2454] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a1d3564b3b40de9e4e4724bfa59d65 in term 1.
I20240501 19:56:08.833873 5863 raft_consensus.cc:2454] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6a1d3564b3b40de9e4e4724bfa59d65 in term 1.
I20240501 19:56:08.833993 6062 leader_election.cc:302] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:08.834874 7120 raft_consensus.cc:2781] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:08.835003 7120 raft_consensus.cc:686] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 LEADER]: Becoming Leader. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Running, Role: LEADER
I20240501 19:56:08.835134 7120 consensus_queue.cc:237] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } }
I20240501 19:56:08.836827 5711 catalog_manager.cc:5484] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 reported cstate change: term changed from 0 to 1, leader changed from <none> to d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67). New cstate: current_term: 1 leader_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:08.837136 7281 raft_consensus.cc:482] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:08.837226 7281 raft_consensus.cc:504] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:08.837597 7281 leader_election.cc:288] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131), 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011)
I20240501 19:56:08.837993 6259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935a6d3444ea4cdbb6ab26271ac3bcf2" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:08.838121 6259 raft_consensus.cc:2454] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:08.838361 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935a6d3444ea4cdbb6ab26271ac3bcf2" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:08.838472 6523 raft_consensus.cc:2454] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:08.838690 5931 leader_election.cc:302] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:08.838876 7281 raft_consensus.cc:2781] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:08.838943 7281 raft_consensus.cc:482] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:08.838997 7281 raft_consensus.cc:3037] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.839789 7281 raft_consensus.cc:504] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:08.840121 7281 leader_election.cc:288] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Requested vote from peers fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131), 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011)
I20240501 19:56:08.840391 6259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935a6d3444ea4cdbb6ab26271ac3bcf2" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:08.840524 6259 raft_consensus.cc:3037] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.840505 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "935a6d3444ea4cdbb6ab26271ac3bcf2" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:08.840626 6523 raft_consensus.cc:3037] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:08.841326 6259 raft_consensus.cc:2454] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:08.841482 6523 raft_consensus.cc:2454] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:08.841576 5931 leader_election.cc:302] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:08.841814 7281 raft_consensus.cc:2781] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:08.841898 7281 raft_consensus.cc:686] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [term 1 LEADER]: Becoming Leader. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Running, Role: LEADER
I20240501 19:56:08.842025 7281 consensus_queue.cc:237] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } }
I20240501 19:56:08.843465 5711 catalog_manager.cc:5484] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3489b69a0428491e957e46effb9b1146 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "3489b69a0428491e957e46effb9b1146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:09.082300 7231 raft_consensus.cc:482] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:09.082432 7231 raft_consensus.cc:504] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:09.082823 7231 leader_election.cc:288] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:09.083184 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f65dcb7f9e43d6959a9cf91f6e51c7" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:09.083245 6391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f65dcb7f9e43d6959a9cf91f6e51c7" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975" is_pre_election: true
I20240501 19:56:09.083365 6127 raft_consensus.cc:2454] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:09.083387 6391 raft_consensus.cc:2454] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 0.
I20240501 19:56:09.083688 5798 leader_election.cc:302] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:09.083868 7231 raft_consensus.cc:2781] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:09.083933 7231 raft_consensus.cc:482] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:09.083978 7231 raft_consensus.cc:3037] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.084933 7231 raft_consensus.cc:504] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:09.085250 7231 leader_election.cc:288] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Requested vote from peers d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087), a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69:42511)
I20240501 19:56:09.085559 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f65dcb7f9e43d6959a9cf91f6e51c7" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:09.085604 6391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7f65dcb7f9e43d6959a9cf91f6e51c7" candidate_uuid: "4629e756401746a8bf4652f995bddb44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2a04e6cd48541839ce83dc1d1790975"
I20240501 19:56:09.085671 6127 raft_consensus.cc:3037] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.085698 6391 raft_consensus.cc:3037] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.086436 6391 raft_consensus.cc:2454] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:09.086449 6127 raft_consensus.cc:2454] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4629e756401746a8bf4652f995bddb44 in term 1.
I20240501 19:56:09.086711 5798 leader_election.cc:302] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:09.086900 7231 raft_consensus.cc:2781] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:09.087014 7231 raft_consensus.cc:686] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [term 1 LEADER]: Becoming Leader. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Running, Role: LEADER
I20240501 19:56:09.087160 7231 consensus_queue.cc:237] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } }
I20240501 19:56:09.090071 5711 catalog_manager.cc:5484] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4629e756401746a8bf4652f995bddb44 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "4629e756401746a8bf4652f995bddb44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:09.207911 7301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:09.208145 7301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:09.211056 7301 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240501 19:56:09.309368 6523 raft_consensus.cc:1261] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.310122 6259 raft_consensus.cc:1261] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.310515 6626 consensus_queue.cc:1035] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.310932 6626 consensus_queue.cc:1035] T 935a6d3444ea4cdbb6ab26271ac3bcf2 P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.313416 6391 raft_consensus.cc:1261] T 1db84f3b4048482f9949a97861defc33 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer d6a1d3564b3b40de9e4e4724bfa59d65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.313935 7297 consensus_queue.cc:1035] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.314007 5863 raft_consensus.cc:1261] T 1db84f3b4048482f9949a97861defc33 P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Refusing update from remote peer d6a1d3564b3b40de9e4e4724bfa59d65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.314592 7297 consensus_queue.cc:1035] T 1db84f3b4048482f9949a97861defc33 P d6a1d3564b3b40de9e4e4724bfa59d65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.324849 6127 raft_consensus.cc:1261] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.325892 6392 raft_consensus.cc:1261] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Refusing update from remote peer 4629e756401746a8bf4652f995bddb44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.326311 7231 consensus_queue.cc:1035] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.327613 6622 consensus_queue.cc:1035] T e7f65dcb7f9e43d6959a9cf91f6e51c7 P 4629e756401746a8bf4652f995bddb44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 135.809 ms
time per row: 0.0679044 ms
I20240501 19:56:09.393651 5720 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39456:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:09.399359 6239 tablet_service.cc:1467] Processing CreateTablet for tablet 8052db84e8894ef9a598246a4797bca9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3266e3afb95e4da7b9bfda677f905bd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:09.399405 5975 tablet_service.cc:1467] Processing CreateTablet for tablet 8052db84e8894ef9a598246a4797bca9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3266e3afb95e4da7b9bfda677f905bd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:09.399689 6239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8052db84e8894ef9a598246a4797bca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.399694 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8052db84e8894ef9a598246a4797bca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.399937 5843 tablet_service.cc:1467] Processing CreateTablet for tablet be00174c139c47159d1b4405529d071d (DEFAULT_TABLE table=rebalance_test_table_11 [id=3266e3afb95e4da7b9bfda677f905bd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:09.400192 5843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be00174c139c47159d1b4405529d071d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.401644 6240 tablet_service.cc:1467] Processing CreateTablet for tablet 2d290e24d76b44a2b1a629f29feab67d (DEFAULT_TABLE table=rebalance_test_table_11 [id=3266e3afb95e4da7b9bfda677f905bd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:09.401811 6240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d290e24d76b44a2b1a629f29feab67d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.402241 7367 tablet_bootstrap.cc:492] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:09.402377 7368 tablet_bootstrap.cc:492] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44: Bootstrap starting.
I20240501 19:56:09.402817 6503 tablet_service.cc:1467] Processing CreateTablet for tablet 8052db84e8894ef9a598246a4797bca9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=3266e3afb95e4da7b9bfda677f905bd3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:09.403084 6503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8052db84e8894ef9a598246a4797bca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.403447 6107 tablet_service.cc:1467] Processing CreateTablet for tablet be00174c139c47159d1b4405529d071d (DEFAULT_TABLE table=rebalance_test_table_11 [id=3266e3afb95e4da7b9bfda677f905bd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:09.403734 6107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be00174c139c47159d1b4405529d071d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.403795 6372 tablet_service.cc:1467] Processing CreateTablet for tablet be00174c139c47159d1b4405529d071d (DEFAULT_TABLE table=rebalance_test_table_11 [id=3266e3afb95e4da7b9bfda677f905bd3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:09.404042 6372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be00174c139c47159d1b4405529d071d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.404088 7366 tablet_bootstrap.cc:492] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:09.404986 7366 tablet_bootstrap.cc:654] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.405937 7370 tablet_bootstrap.cc:492] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65: Bootstrap starting.
I20240501 19:56:09.406270 7368 tablet_bootstrap.cc:654] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.406594 6504 tablet_service.cc:1467] Processing CreateTablet for tablet 2d290e24d76b44a2b1a629f29feab67d (DEFAULT_TABLE table=rebalance_test_table_11 [id=3266e3afb95e4da7b9bfda677f905bd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:09.406682 7370 tablet_bootstrap.cc:654] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.406848 6504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d290e24d76b44a2b1a629f29feab67d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.407119 7371 tablet_bootstrap.cc:492] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975: Bootstrap starting.
I20240501 19:56:09.407624 7370 tablet_bootstrap.cc:492] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65: No bootstrap required, opened a new log
I20240501 19:56:09.407686 7370 ts_tablet_manager.cc:1367] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:09.408023 7371 tablet_bootstrap.cc:654] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.408064 7370 raft_consensus.cc:348] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:09.408154 7370 raft_consensus.cc:374] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.408188 7370 raft_consensus.cc:729] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1d3564b3b40de9e4e4724bfa59d65, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.408320 7370 consensus_queue.cc:260] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:09.408507 5975 tablet_service.cc:1467] Processing CreateTablet for tablet 2d290e24d76b44a2b1a629f29feab67d (DEFAULT_TABLE table=rebalance_test_table_11 [id=3266e3afb95e4da7b9bfda677f905bd3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:09.408592 7370 ts_tablet_manager.cc:1398] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:09.408759 5975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d290e24d76b44a2b1a629f29feab67d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:09.409749 7371 tablet_bootstrap.cc:492] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975: No bootstrap required, opened a new log
I20240501 19:56:09.409834 7371 ts_tablet_manager.cc:1367] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:09.410254 7371 raft_consensus.cc:348] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:09.410351 7371 raft_consensus.cc:374] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.410387 7371 raft_consensus.cc:729] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.410506 7371 consensus_queue.cc:260] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:09.410753 7371 ts_tablet_manager.cc:1398] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:09.411653 7375 tablet_bootstrap.cc:492] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:09.412492 7375 tablet_bootstrap.cc:654] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.413674 7375 tablet_bootstrap.cc:492] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:09.413749 7375 ts_tablet_manager.cc:1367] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:09.414137 7375 raft_consensus.cc:348] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:09.414235 7375 raft_consensus.cc:374] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.414263 7375 raft_consensus.cc:729] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.414363 7375 consensus_queue.cc:260] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:09.414584 7375 ts_tablet_manager.cc:1398] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:09.414716 7375 tablet_bootstrap.cc:492] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56: Bootstrap starting.
I20240501 19:56:09.414925 7367 tablet_bootstrap.cc:654] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.415459 7375 tablet_bootstrap.cc:654] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.416277 7368 tablet_bootstrap.cc:492] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44: No bootstrap required, opened a new log
I20240501 19:56:09.416385 7366 tablet_bootstrap.cc:492] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:09.416410 7368 ts_tablet_manager.cc:1367] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20240501 19:56:09.416393 7189 raft_consensus.cc:482] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:09.416469 7366 ts_tablet_manager.cc:1367] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:56:09.416471 7189 raft_consensus.cc:504] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:09.416854 7189 leader_election.cc:288] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:09.416810 7368 raft_consensus.cc:348] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:09.416853 7366 raft_consensus.cc:348] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:09.416913 7368 raft_consensus.cc:374] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.416939 7366 raft_consensus.cc:374] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.416944 7368 raft_consensus.cc:729] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4629e756401746a8bf4652f995bddb44, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.416966 7366 raft_consensus.cc:729] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.417095 7366 consensus_queue.cc:260] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:09.417145 7368 consensus_queue.cc:260] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:09.417366 7366 ts_tablet_manager.cc:1398] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:09.417348 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be00174c139c47159d1b4405529d071d" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" is_pre_election: true
I20240501 19:56:09.417441 7368 ts_tablet_manager.cc:1398] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:09.417462 6127 raft_consensus.cc:2454] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:09.417719 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be00174c139c47159d1b4405529d071d" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44" is_pre_election: true
I20240501 19:56:09.417762 6327 leader_election.cc:302] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2a04e6cd48541839ce83dc1d1790975, d6a1d3564b3b40de9e4e4724bfa59d65; no voters:
I20240501 19:56:09.417811 5863 raft_consensus.cc:2454] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 0.
I20240501 19:56:09.417918 7189 raft_consensus.cc:2781] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:09.417959 7189 raft_consensus.cc:482] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:09.417994 7189 raft_consensus.cc:3037] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.418658 7366 tablet_bootstrap.cc:492] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146: Bootstrap starting.
I20240501 19:56:09.418988 7189 raft_consensus.cc:504] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:09.419351 7189 leader_election.cc:288] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Requested vote from peers 4629e756401746a8bf4652f995bddb44 (127.5.139.65:40845), d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087)
I20240501 19:56:09.419618 7366 tablet_bootstrap.cc:654] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.419948 5863 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be00174c139c47159d1b4405529d071d" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4629e756401746a8bf4652f995bddb44"
I20240501 19:56:09.420045 5863 raft_consensus.cc:3037] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.421087 5863 raft_consensus.cc:2454] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:09.421176 7366 tablet_bootstrap.cc:492] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146: No bootstrap required, opened a new log
I20240501 19:56:09.421262 7366 ts_tablet_manager.cc:1367] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:09.421397 6328 leader_election.cc:302] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4629e756401746a8bf4652f995bddb44, a2a04e6cd48541839ce83dc1d1790975; no voters:
I20240501 19:56:09.421561 7189 raft_consensus.cc:2781] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:09.421658 7189 raft_consensus.cc:686] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [term 1 LEADER]: Becoming Leader. State: Replica: a2a04e6cd48541839ce83dc1d1790975, State: Running, Role: LEADER
I20240501 19:56:09.421736 7366 raft_consensus.cc:348] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:09.421839 7366 raft_consensus.cc:374] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.421784 7189 consensus_queue.cc:237] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } }
I20240501 19:56:09.421873 7366 raft_consensus.cc:729] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.421972 7366 consensus_queue.cc:260] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:09.422246 7366 ts_tablet_manager.cc:1398] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:09.422863 7375 tablet_bootstrap.cc:492] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56: No bootstrap required, opened a new log
I20240501 19:56:09.422878 6127 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be00174c139c47159d1b4405529d071d" candidate_uuid: "a2a04e6cd48541839ce83dc1d1790975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65"
I20240501 19:56:09.422941 7375 ts_tablet_manager.cc:1367] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:09.422999 6127 raft_consensus.cc:3037] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.423403 7375 raft_consensus.cc:348] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:09.423508 7375 raft_consensus.cc:374] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.423548 7375 raft_consensus.cc:729] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.423660 7375 consensus_queue.cc:260] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:09.423945 7375 ts_tablet_manager.cc:1398] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:09.423970 6127 raft_consensus.cc:2454] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2a04e6cd48541839ce83dc1d1790975 in term 1.
I20240501 19:56:09.424777 5716 catalog_manager.cc:5484] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2a04e6cd48541839ce83dc1d1790975 (127.5.139.69). New cstate: current_term: 1 leader_uuid: "a2a04e6cd48541839ce83dc1d1790975" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2a04e6cd48541839ce83dc1d1790975" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42511 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:09.429783 7367 tablet_bootstrap.cc:492] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:09.429888 7367 ts_tablet_manager.cc:1367] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.000s
I20240501 19:56:09.430254 7367 raft_consensus.cc:348] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:09.430351 7367 raft_consensus.cc:374] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.430385 7367 raft_consensus.cc:729] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.430521 7367 consensus_queue.cc:260] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:09.430814 7367 ts_tablet_manager.cc:1398] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:09.431035 7367 tablet_bootstrap.cc:492] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd: Bootstrap starting.
I20240501 19:56:09.431797 7367 tablet_bootstrap.cc:654] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:09.432760 7367 tablet_bootstrap.cc:492] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd: No bootstrap required, opened a new log
I20240501 19:56:09.432843 7367 ts_tablet_manager.cc:1367] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:09.433240 7367 raft_consensus.cc:348] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:09.433323 7367 raft_consensus.cc:374] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:09.433357 7367 raft_consensus.cc:729] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb1af421bf6a480587c69697af9f39dd, State: Initialized, Role: FOLLOWER
I20240501 19:56:09.433481 7367 consensus_queue.cc:260] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:09.433689 7367 ts_tablet_manager.cc:1398] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:09.505627 6600 raft_consensus.cc:482] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:09.505786 6600 raft_consensus.cc:504] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:09.506165 6600 leader_election.cc:288] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:09.506551 6259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8052db84e8894ef9a598246a4797bca9" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:09.506570 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8052db84e8894ef9a598246a4797bca9" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146" is_pre_election: true
I20240501 19:56:09.506675 6259 raft_consensus.cc:2454] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:09.506753 5994 raft_consensus.cc:2454] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 0.
I20240501 19:56:09.506969 6460 leader_election.cc:302] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 60b2c1ef2e86405bba90eaa1db142e56, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:09.507161 6600 raft_consensus.cc:2781] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:09.507229 6600 raft_consensus.cc:482] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:09.507257 6600 raft_consensus.cc:3037] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.508060 6600 raft_consensus.cc:504] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:09.508404 6600 leader_election.cc:288] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Requested vote from peers fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131), 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835)
I20240501 19:56:09.508697 6259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8052db84e8894ef9a598246a4797bca9" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:09.508769 5994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8052db84e8894ef9a598246a4797bca9" candidate_uuid: "60b2c1ef2e86405bba90eaa1db142e56" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3489b69a0428491e957e46effb9b1146"
I20240501 19:56:09.508822 6259 raft_consensus.cc:3037] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.508903 5994 raft_consensus.cc:3037] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.509688 6259 raft_consensus.cc:2454] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:09.509894 5994 raft_consensus.cc:2454] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 60b2c1ef2e86405bba90eaa1db142e56 in term 1.
I20240501 19:56:09.510186 6459 leader_election.cc:302] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:09.510354 6600 raft_consensus.cc:2781] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:09.510545 6600 raft_consensus.cc:686] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 LEADER]: Becoming Leader. State: Replica: 60b2c1ef2e86405bba90eaa1db142e56, State: Running, Role: LEADER
I20240501 19:56:09.510684 6600 consensus_queue.cc:237] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } }
I20240501 19:56:09.512382 5716 catalog_manager.cc:5484] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 reported cstate change: term changed from 0 to 1, leader changed from <none> to 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70). New cstate: current_term: 1 leader_uuid: "60b2c1ef2e86405bba90eaa1db142e56" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:09.528389 7281 raft_consensus.cc:482] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:09.528506 7281 raft_consensus.cc:504] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:09.528861 7281 leader_election.cc:288] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:09.529238 6259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d290e24d76b44a2b1a629f29feab67d" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd" is_pre_election: true
I20240501 19:56:09.529245 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d290e24d76b44a2b1a629f29feab67d" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56" is_pre_election: true
I20240501 19:56:09.529363 6523 raft_consensus.cc:2454] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:09.529367 6259 raft_consensus.cc:2454] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 0.
I20240501 19:56:09.529624 5930 leader_election.cc:302] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, 60b2c1ef2e86405bba90eaa1db142e56; no voters:
I20240501 19:56:09.529791 7281 raft_consensus.cc:2781] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:09.529839 7281 raft_consensus.cc:482] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:09.529870 7281 raft_consensus.cc:3037] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.530648 7281 raft_consensus.cc:504] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:09.530946 7281 leader_election.cc:288] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Requested vote from peers 60b2c1ef2e86405bba90eaa1db142e56 (127.5.139.70:35011), fb1af421bf6a480587c69697af9f39dd (127.5.139.68:39131)
I20240501 19:56:09.531195 6523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d290e24d76b44a2b1a629f29feab67d" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "60b2c1ef2e86405bba90eaa1db142e56"
I20240501 19:56:09.531260 6259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d290e24d76b44a2b1a629f29feab67d" candidate_uuid: "3489b69a0428491e957e46effb9b1146" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb1af421bf6a480587c69697af9f39dd"
I20240501 19:56:09.531307 6523 raft_consensus.cc:3037] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.531353 6259 raft_consensus.cc:3037] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:09.532198 6259 raft_consensus.cc:2454] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:09.532198 6523 raft_consensus.cc:2454] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3489b69a0428491e957e46effb9b1146 in term 1.
I20240501 19:56:09.532419 5931 leader_election.cc:302] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3489b69a0428491e957e46effb9b1146, fb1af421bf6a480587c69697af9f39dd; no voters:
I20240501 19:56:09.532701 7281 raft_consensus.cc:2781] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:09.532811 7281 raft_consensus.cc:686] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [term 1 LEADER]: Becoming Leader. State: Replica: 3489b69a0428491e957e46effb9b1146, State: Running, Role: LEADER
I20240501 19:56:09.532956 7281 consensus_queue.cc:237] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } }
I20240501 19:56:09.534791 5716 catalog_manager.cc:5484] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3489b69a0428491e957e46effb9b1146 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "3489b69a0428491e957e46effb9b1146" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:09.672224 7385 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:09.672461 7385 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:09.675462 7385 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240501 19:56:09.747984 6523 raft_consensus.cc:1261] T 2d290e24d76b44a2b1a629f29feab67d P 60b2c1ef2e86405bba90eaa1db142e56 [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.749212 7281 consensus_queue.cc:1035] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "60b2c1ef2e86405bba90eaa1db142e56" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 35011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.749557 6259 raft_consensus.cc:1261] T 2d290e24d76b44a2b1a629f29feab67d P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 3489b69a0428491e957e46effb9b1146: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.749922 5994 raft_consensus.cc:1261] T 8052db84e8894ef9a598246a4797bca9 P 3489b69a0428491e957e46effb9b1146 [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.750461 6600 consensus_queue.cc:1035] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3489b69a0428491e957e46effb9b1146" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 33835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.752506 6626 consensus_queue.cc:1035] T 2d290e24d76b44a2b1a629f29feab67d P 3489b69a0428491e957e46effb9b1146 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.754623 6259 raft_consensus.cc:1261] T 8052db84e8894ef9a598246a4797bca9 P fb1af421bf6a480587c69697af9f39dd [term 1 FOLLOWER]: Refusing update from remote peer 60b2c1ef2e86405bba90eaa1db142e56: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.755344 6600 consensus_queue.cc:1035] T 8052db84e8894ef9a598246a4797bca9 P 60b2c1ef2e86405bba90eaa1db142e56 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb1af421bf6a480587c69697af9f39dd" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 39131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.765904 6127 raft_consensus.cc:1261] T be00174c139c47159d1b4405529d071d P d6a1d3564b3b40de9e4e4724bfa59d65 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.766263 5863 raft_consensus.cc:1261] T be00174c139c47159d1b4405529d071d P 4629e756401746a8bf4652f995bddb44 [term 1 FOLLOWER]: Refusing update from remote peer a2a04e6cd48541839ce83dc1d1790975: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:09.766860 7324 consensus_queue.cc:1035] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4629e756401746a8bf4652f995bddb44" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 40845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:09.766996 7189 consensus_queue.cc:1035] T be00174c139c47159d1b4405529d071d P a2a04e6cd48541839ce83dc1d1790975 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1d3564b3b40de9e4e4724bfa59d65" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 103.966 ms
time per row: 0.0519829 ms
I20240501 19:56:10.030442 5843 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:10.031008 5975 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.032095 6503 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.037943 6240 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.042646 6372 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:10.043337 6107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.163911 5975 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.173205 6372 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:10.175060 6107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.175581 6503 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.175621 5843 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:10.200322 6240 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.320521 5843 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:10.321185 6503 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.321650 5975 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.322867 6372 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:10.330299 6107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.334431 6240 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.451310 5843 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:10.451380 6107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.453102 6240 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.460148 6503 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.466408 5975 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.470618 6372 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:10.584287 5975 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.588724 6240 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.590680 6372 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:10.594108 6503 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.594399 5843 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:10.621426 6107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.734318 5843 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:10.736892 6503 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.738386 6240 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.738890 5975 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:10.742739 6107 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:10.745208 6372 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
4a381a255c7d46878094e496e9d7ef5a | 127.5.139.126:35927 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.139.84:42691 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
3489b69a0428491e957e46effb9b1146 | 127.5.139.66:33835 | HEALTHY | /A | 6 | 0
4629e756401746a8bf4652f995bddb44 | 127.5.139.65:40845 | HEALTHY | /A | 9 | 0
60b2c1ef2e86405bba90eaa1db142e56 | 127.5.139.70:35011 | HEALTHY | /C | 6 | 0
a2a04e6cd48541839ce83dc1d1790975 | 127.5.139.69:42511 | HEALTHY | /C | 7 | 0
d6a1d3564b3b40de9e4e4724bfa59d65 | 127.5.139.67:36087 | HEALTHY | /B | 2 | 0
fb1af421bf6a480587c69697af9f39dd | 127.5.139.68:39131 | HEALTHY | /B | 6 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/B | 2
/A | 2
/C | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.5.139.65 | experimental | 127.5.139.65:40845
local_ip_for_outbound_sockets | 127.5.139.66 | experimental | 127.5.139.66:33835
local_ip_for_outbound_sockets | 127.5.139.67 | experimental | 127.5.139.67:36087
local_ip_for_outbound_sockets | 127.5.139.68 | experimental | 127.5.139.68:39131
local_ip_for_outbound_sockets | 127.5.139.69 | experimental | 127.5.139.69:42511
local_ip_for_outbound_sockets | 127.5.139.70 | experimental | 127.5.139.70:35011
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.139.65:40845
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.139.66:33835
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.139.67:36087
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.139.68:39131
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.139.69:42511
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.139.70:35011
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.139.84:42691 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 19:56:10.787061 5677 log_verifier.cc:126] Checking tablet 0609d499c8d3425697918f47a660d403
I20240501 19:56:10.815771 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0609d499c8d3425697918f47a660d403
I20240501 19:56:10.815861 5677 log_verifier.cc:126] Checking tablet 07fd0c6dc02740469cfa36ee5ef587a0
I20240501 19:56:10.840293 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 07fd0c6dc02740469cfa36ee5ef587a0
I20240501 19:56:10.840360 5677 log_verifier.cc:126] Checking tablet 195246eb76cf4aa294aad0b658588a89
I20240501 19:56:10.864249 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 195246eb76cf4aa294aad0b658588a89
I20240501 19:56:10.864317 5677 log_verifier.cc:126] Checking tablet 1bc7c03eb2f447eb9d268b4cc7036dc3
I20240501 19:56:10.888144 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1bc7c03eb2f447eb9d268b4cc7036dc3
I20240501 19:56:10.888221 5677 log_verifier.cc:126] Checking tablet 1db84f3b4048482f9949a97861defc33
I20240501 19:56:10.913558 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1db84f3b4048482f9949a97861defc33
I20240501 19:56:10.913619 5677 log_verifier.cc:126] Checking tablet 2d290e24d76b44a2b1a629f29feab67d
I20240501 19:56:10.938021 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 2d290e24d76b44a2b1a629f29feab67d
I20240501 19:56:10.938092 5677 log_verifier.cc:126] Checking tablet 3601c1f90c8f40f8ae4515bad00d412f
I20240501 19:56:10.962142 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3601c1f90c8f40f8ae4515bad00d412f
I20240501 19:56:10.962226 5677 log_verifier.cc:126] Checking tablet 3bc35d49716049c0aaf3407d786d1100
I20240501 19:56:10.986546 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3bc35d49716049c0aaf3407d786d1100
I20240501 19:56:10.986605 5677 log_verifier.cc:126] Checking tablet 46e06196fcff476fb8e75fae159a6b42
I20240501 19:56:11.010996 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 46e06196fcff476fb8e75fae159a6b42
I20240501 19:56:11.011085 5677 log_verifier.cc:126] Checking tablet 48978f6f1980497cb3bd91c26de554d6
I20240501 19:56:11.035871 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 48978f6f1980497cb3bd91c26de554d6
I20240501 19:56:11.035938 5677 log_verifier.cc:126] Checking tablet 607e3347b7934e08a169f1a187941f6b
I20240501 19:56:11.060696 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 607e3347b7934e08a169f1a187941f6b
I20240501 19:56:11.060762 5677 log_verifier.cc:126] Checking tablet 631040a5e0114513b578a6794a69bab5
I20240501 19:56:11.085456 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 631040a5e0114513b578a6794a69bab5
I20240501 19:56:11.085522 5677 log_verifier.cc:126] Checking tablet 7112210d92834e2c96c2f6ffe614ec98
I20240501 19:56:11.109097 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7112210d92834e2c96c2f6ffe614ec98
I20240501 19:56:11.109163 5677 log_verifier.cc:126] Checking tablet 747ac8627fc94390b942717801d9e61f
I20240501 19:56:11.134316 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 747ac8627fc94390b942717801d9e61f
I20240501 19:56:11.134378 5677 log_verifier.cc:126] Checking tablet 77ce1fa3ae5942e08cac2dad70319167
I20240501 19:56:11.158417 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 77ce1fa3ae5942e08cac2dad70319167
I20240501 19:56:11.158488 5677 log_verifier.cc:126] Checking tablet 8052db84e8894ef9a598246a4797bca9
I20240501 19:56:11.182507 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8052db84e8894ef9a598246a4797bca9
I20240501 19:56:11.182574 5677 log_verifier.cc:126] Checking tablet 8aede5dd927f4029ad1c9d77dcf073a0
I20240501 19:56:11.211755 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8aede5dd927f4029ad1c9d77dcf073a0
I20240501 19:56:11.211817 5677 log_verifier.cc:126] Checking tablet 8bdb29bb3e4b4cf1ad2848bac08c83d0
I20240501 19:56:11.236430 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8bdb29bb3e4b4cf1ad2848bac08c83d0
I20240501 19:56:11.236491 5677 log_verifier.cc:126] Checking tablet 8d61f765fc274f2d841c0843ec8cc1fa
I20240501 19:56:11.260586 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8d61f765fc274f2d841c0843ec8cc1fa
I20240501 19:56:11.260658 5677 log_verifier.cc:126] Checking tablet 935a6d3444ea4cdbb6ab26271ac3bcf2
I20240501 19:56:11.284695 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 935a6d3444ea4cdbb6ab26271ac3bcf2
I20240501 19:56:11.284767 5677 log_verifier.cc:126] Checking tablet 980bc7bedd7d48d083b2c0b951339289
I20240501 19:56:11.309012 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 980bc7bedd7d48d083b2c0b951339289
I20240501 19:56:11.309105 5677 log_verifier.cc:126] Checking tablet 9fb5dbfc92754524910519306c02b132
I20240501 19:56:11.333631 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9fb5dbfc92754524910519306c02b132
I20240501 19:56:11.333705 5677 log_verifier.cc:126] Checking tablet a1aa22f900e54f1dbdc735b7aa2377e6
I20240501 19:56:11.360167 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a1aa22f900e54f1dbdc735b7aa2377e6
I20240501 19:56:11.360232 5677 log_verifier.cc:126] Checking tablet a6161b2d95084e9c90fb83929d2d1c70
I20240501 19:56:11.383980 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a6161b2d95084e9c90fb83929d2d1c70
I20240501 19:56:11.384047 5677 log_verifier.cc:126] Checking tablet b3c88941cbe044e8954d560f52b0695d
I20240501 19:56:11.408700 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b3c88941cbe044e8954d560f52b0695d
I20240501 19:56:11.408776 5677 log_verifier.cc:126] Checking tablet be00174c139c47159d1b4405529d071d
I20240501 19:56:11.434971 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet be00174c139c47159d1b4405529d071d
I20240501 19:56:11.435036 5677 log_verifier.cc:126] Checking tablet c1e625cabd1845bda82e7b78317fe081
I20240501 19:56:11.459692 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c1e625cabd1845bda82e7b78317fe081
I20240501 19:56:11.459761 5677 log_verifier.cc:126] Checking tablet c37bf45daf71453c816e5247765220e4
I20240501 19:56:11.484129 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c37bf45daf71453c816e5247765220e4
I20240501 19:56:11.484200 5677 log_verifier.cc:126] Checking tablet cec70d5cb2c64222ad47ecc922be3da6
I20240501 19:56:11.508213 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cec70d5cb2c64222ad47ecc922be3da6
I20240501 19:56:11.508308 5677 log_verifier.cc:126] Checking tablet d30c0fccb70845978605837176173e6f
I20240501 19:56:11.536088 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d30c0fccb70845978605837176173e6f
I20240501 19:56:11.536160 5677 log_verifier.cc:126] Checking tablet e17e4c0e86564147908d01602f82480a
I20240501 19:56:11.560787 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e17e4c0e86564147908d01602f82480a
I20240501 19:56:11.560850 5677 log_verifier.cc:126] Checking tablet e7f65dcb7f9e43d6959a9cf91f6e51c7
I20240501 19:56:11.585141 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e7f65dcb7f9e43d6959a9cf91f6e51c7
I20240501 19:56:11.585201 5677 log_verifier.cc:126] Checking tablet ecdd31a4e11847e1b747d5c4f6fc409f
I20240501 19:56:11.608381 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ecdd31a4e11847e1b747d5c4f6fc409f
I20240501 19:56:11.608462 5677 log_verifier.cc:126] Checking tablet ed54775aa8764fe18974093dcbb7d49b
I20240501 19:56:11.632534 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ed54775aa8764fe18974093dcbb7d49b
I20240501 19:56:11.632615 5677 log_verifier.cc:126] Checking tablet ee048890b02446b1a7cdd28b3a54e223
I20240501 19:56:11.656242 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet ee048890b02446b1a7cdd28b3a54e223
I20240501 19:56:11.656316 5677 log_verifier.cc:126] Checking tablet f41918d1c0094ec9aa4b6d1b8c57b9d2
I20240501 19:56:11.679811 5677 log_verifier.cc:177] Verified matching terms for 3 ops in tablet f41918d1c0094ec9aa4b6d1b8c57b9d2
I20240501 19:56:11.693127 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 5781
I20240501 19:56:11.701411 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 5912
I20240501 19:56:11.708065 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 6044
I20240501 19:56:11.714378 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 6177
W20240501 19:56:11.716514 6327 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.67:36087: connect: Connection refused (error 111)
W20240501 19:56:11.717758 6327 consensus_peers.cc:485] T 0609d499c8d3425697918f47a660d403 P a2a04e6cd48541839ce83dc1d1790975 -> Peer d6a1d3564b3b40de9e4e4724bfa59d65 (127.5.139.67:36087): Couldn't send request to peer d6a1d3564b3b40de9e4e4724bfa59d65. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.67:36087: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:11.717973 6459 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.66:33835: connect: Connection refused (error 111)
W20240501 19:56:11.718822 6459 consensus_peers.cc:485] T 631040a5e0114513b578a6794a69bab5 P 60b2c1ef2e86405bba90eaa1db142e56 -> Peer 3489b69a0428491e957e46effb9b1146 (127.5.139.66:33835): Couldn't send request to peer 3489b69a0428491e957e46effb9b1146. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.66:33835: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:11.721127 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 6309
I20240501 19:56:11.727512 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 6441
I20240501 19:56:11.734187 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 5689
2024-05-01T19:56:11Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (13643 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (13643 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240501 19:56:11.851683 5677 test_util.cc:268] Using random seed: -522384482
I20240501 19:56:11.852422 5677 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:56:11.852473 5677 ts_itest-base.cc:113] --------------
I20240501 19:56:11.852485 5677 ts_itest-base.cc:114] 5 tablet servers
I20240501 19:56:11.852495 5677 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:56:11.852519 5677 ts_itest-base.cc:116] --------------
2024-05-01T19:56:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:11Z Disabled control of system clock
I20240501 19:56:11.860642 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.139.126:44591
--webserver_interface=127.5.139.126
--webserver_port=0
--builtin_ntp_servers=127.5.139.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.139.126:44591
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 19:56:11.967320 7712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:11.967588 7712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:11.967664 7712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:11.970752 7712 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:11.970830 7712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:11.970867 7712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:11.970911 7712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:11.970952 7712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:11.974761 7712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.139.126:44591
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.139.126:44591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.139.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.7712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:11.975649 7712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:11.976752 7712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:11.984541 7717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:11.985251 7720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:11.986007 7712 server_base.cc:1034] running on GCE node
W20240501 19:56:11.986562 7718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:11.987084 7712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:11.988003 7712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:11.989091 7712 hybrid_clock.cc:648] HybridClock initialized: now 1714593371989071 us; error 37 us; skew 500 ppm
I20240501 19:56:11.990919 7712 webserver.cc:460] Webserver started at http://127.5.139.126:33827/ using document root <none> and password file <none>
I20240501 19:56:11.991456 7712 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:11.991511 7712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:11.991680 7712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:11.993351 7712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f02ce989c79942a5b77d5aa55f906ddc"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-twc7"
I20240501 19:56:11.993819 7712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f02ce989c79942a5b77d5aa55f906ddc"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-twc7"
I20240501 19:56:11.997095 7712 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20240501 19:56:11.999190 7726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.000135 7712 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:12.000254 7712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f02ce989c79942a5b77d5aa55f906ddc"
format_stamp: "Formatted at 2024-05-01 19:56:11 on dist-test-slave-twc7"
I20240501 19:56:12.000334 7712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:12.013326 7712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:12.013947 7712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:12.014107 7712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:12.021027 7778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.126:44591 every 8 connection(s)
I20240501 19:56:12.021025 7712 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.126:44591
I20240501 19:56:12.022049 7712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:56:12.024788 7779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:12.025619 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 7712
I20240501 19:56:12.025722 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:56:12.030681 7779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc: Bootstrap starting.
I20240501 19:56:12.032857 7779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:12.033682 7779 log.cc:824] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:12.035364 7779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc: No bootstrap required, opened a new log
I20240501 19:56:12.038002 7779 raft_consensus.cc:348] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02ce989c79942a5b77d5aa55f906ddc" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 44591 } }
I20240501 19:56:12.038182 7779 raft_consensus.cc:374] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:12.038218 7779 raft_consensus.cc:729] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02ce989c79942a5b77d5aa55f906ddc, State: Initialized, Role: FOLLOWER
I20240501 19:56:12.038566 7779 consensus_queue.cc:260] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02ce989c79942a5b77d5aa55f906ddc" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 44591 } }
I20240501 19:56:12.038681 7779 raft_consensus.cc:388] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:12.038717 7779 raft_consensus.cc:482] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:12.038787 7779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:12.046451 7779 raft_consensus.cc:504] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02ce989c79942a5b77d5aa55f906ddc" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 44591 } }
I20240501 19:56:12.046831 7779 leader_election.cc:302] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f02ce989c79942a5b77d5aa55f906ddc; no voters:
I20240501 19:56:12.047111 7779 leader_election.cc:288] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:12.047236 7784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:12.047449 7784 raft_consensus.cc:686] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [term 1 LEADER]: Becoming Leader. State: Replica: f02ce989c79942a5b77d5aa55f906ddc, State: Running, Role: LEADER
I20240501 19:56:12.047778 7784 consensus_queue.cc:237] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02ce989c79942a5b77d5aa55f906ddc" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 44591 } }
I20240501 19:56:12.048243 7779 sys_catalog.cc:564] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:12.049491 7786 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f02ce989c79942a5b77d5aa55f906ddc. Latest consensus state: current_term: 1 leader_uuid: "f02ce989c79942a5b77d5aa55f906ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02ce989c79942a5b77d5aa55f906ddc" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 44591 } } }
I20240501 19:56:12.049523 7785 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f02ce989c79942a5b77d5aa55f906ddc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f02ce989c79942a5b77d5aa55f906ddc" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 44591 } } }
I20240501 19:56:12.049609 7786 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:12.049644 7785 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:12.049973 7788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:12.053293 7788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:12.058643 7788 catalog_manager.cc:1347] Generated new cluster ID: 1b1248c8a37d44eab49b6765ff1e082b
I20240501 19:56:12.058745 7788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:12.074767 7788 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:12.075949 7788 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:12.085239 7788 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f02ce989c79942a5b77d5aa55f906ddc: Generated new TSK 0
I20240501 19:56:12.085995 7788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:12.099052 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.65:0
--local_ip_for_outbound_sockets=127.5.139.65
--webserver_interface=127.5.139.65
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--builtin_ntp_servers=127.5.139.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:12.211967 7803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:12.212191 7803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:12.212273 7803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:12.215575 7803 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:12.215662 7803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:12.215770 7803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.65
I20240501 19:56:12.219398 7803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.139.65
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--never_fsync=true
--heap_profile_path=/tmp/kudu.7803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.65
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:12.220327 7803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:12.221565 7803 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:12.231165 7808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.231751 7809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.233179 7811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:12.233230 7803 server_base.cc:1034] running on GCE node
I20240501 19:56:12.233645 7803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:12.234261 7803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:12.235426 7803 hybrid_clock.cc:648] HybridClock initialized: now 1714593372235401 us; error 39 us; skew 500 ppm
I20240501 19:56:12.237485 7803 webserver.cc:460] Webserver started at http://127.5.139.65:40793/ using document root <none> and password file <none>
I20240501 19:56:12.238034 7803 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:12.238087 7803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:12.238274 7803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:12.239903 7803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9682d399a91e4df8ade08ee2f96692b8"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.240358 7803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9682d399a91e4df8ade08ee2f96692b8"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.243626 7803 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:12.246167 7817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.247133 7803 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240501 19:56:12.247259 7803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9682d399a91e4df8ade08ee2f96692b8"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.247359 7803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:12.276515 7803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:12.277209 7803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:12.277379 7803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:12.277935 7803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:12.279242 7803 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:12.279294 7803 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.279356 7803 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:12.279384 7803 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.287704 7803 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.65:40735
I20240501 19:56:12.287741 7930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.65:40735 every 8 connection(s)
I20240501 19:56:12.288614 7803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:56:12.295243 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 7803
I20240501 19:56:12.295377 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:56:12.297533 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.66:0
--local_ip_for_outbound_sockets=127.5.139.66
--webserver_interface=127.5.139.66
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--builtin_ntp_servers=127.5.139.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:12.297540 7931 heartbeater.cc:347] Connected to a master server at 127.5.139.126:44591
I20240501 19:56:12.297765 7931 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:12.298265 7931 heartbeater.cc:511] Master 127.5.139.126:44591 requested a full tablet report, sending...
I20240501 19:56:12.299758 7743 ts_manager.cc:194] Registered new tserver with Master: 9682d399a91e4df8ade08ee2f96692b8 (127.5.139.65:40735)
I20240501 19:56:12.301437 7743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.65:47869
W20240501 19:56:12.414136 7934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:12.414345 7934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:12.414386 7934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:12.417268 7934 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:12.417320 7934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:12.417384 7934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.66
I20240501 19:56:12.420707 7934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.139.66
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--never_fsync=true
--heap_profile_path=/tmp/kudu.7934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.66
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:12.421490 7934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:12.422534 7934 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:12.430948 7942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.431620 7940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.432368 7939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:12.433111 7934 server_base.cc:1034] running on GCE node
I20240501 19:56:12.433571 7934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:12.434116 7934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:12.435271 7934 hybrid_clock.cc:648] HybridClock initialized: now 1714593372435250 us; error 42 us; skew 500 ppm
I20240501 19:56:12.437260 7934 webserver.cc:460] Webserver started at http://127.5.139.66:46791/ using document root <none> and password file <none>
I20240501 19:56:12.437876 7934 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:12.437932 7934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:12.438082 7934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:12.439707 7934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cbdb01e549004ac0a811cb76dc55527d"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.440644 7934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cbdb01e549004ac0a811cb76dc55527d"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.444106 7934 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:12.446327 7948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.447335 7934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:12.447428 7934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cbdb01e549004ac0a811cb76dc55527d"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.447538 7934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:12.468333 7934 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:12.468947 7934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:12.469134 7934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:12.469631 7934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:12.470453 7934 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:12.470499 7934 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.470536 7934 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:12.470557 7934 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.478994 7934 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.66:42941
I20240501 19:56:12.479022 8061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.66:42941 every 8 connection(s)
I20240501 19:56:12.479913 7934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:56:12.483779 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 7934
I20240501 19:56:12.483889 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:56:12.485989 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.67:0
--local_ip_for_outbound_sockets=127.5.139.67
--webserver_interface=127.5.139.67
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--builtin_ntp_servers=127.5.139.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:12.488749 8062 heartbeater.cc:347] Connected to a master server at 127.5.139.126:44591
I20240501 19:56:12.488976 8062 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:12.489537 8062 heartbeater.cc:511] Master 127.5.139.126:44591 requested a full tablet report, sending...
I20240501 19:56:12.490453 7743 ts_manager.cc:194] Registered new tserver with Master: cbdb01e549004ac0a811cb76dc55527d (127.5.139.66:42941)
I20240501 19:56:12.490993 7743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.66:51413
W20240501 19:56:12.591387 8065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:12.591614 8065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:12.591714 8065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:12.594692 8065 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:12.594766 8065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:12.594861 8065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.67
I20240501 19:56:12.598335 8065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.139.67
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--never_fsync=true
--heap_profile_path=/tmp/kudu.8065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.67
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:12.599175 8065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:12.600230 8065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:12.608667 8070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.609361 8071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.610044 8073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:12.610778 8065 server_base.cc:1034] running on GCE node
I20240501 19:56:12.611161 8065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:12.611685 8065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:12.612852 8065 hybrid_clock.cc:648] HybridClock initialized: now 1714593372612843 us; error 32 us; skew 500 ppm
I20240501 19:56:12.614765 8065 webserver.cc:460] Webserver started at http://127.5.139.67:44517/ using document root <none> and password file <none>
I20240501 19:56:12.615311 8065 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:12.615375 8065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:12.615547 8065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:12.617208 8065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3905a8bbb6be48b3912362133e2c6aca"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.617664 8065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3905a8bbb6be48b3912362133e2c6aca"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.620792 8065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:12.622906 8079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.623860 8065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:12.623962 8065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3905a8bbb6be48b3912362133e2c6aca"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.624047 8065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:12.657613 8065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:12.658408 8065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:12.658609 8065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:12.659157 8065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:12.659996 8065 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:12.660043 8065 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.660081 8065 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:12.660102 8065 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.668294 8065 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.67:41301
I20240501 19:56:12.668342 8192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.67:41301 every 8 connection(s)
I20240501 19:56:12.669219 8065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:56:12.671926 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 8065
I20240501 19:56:12.672006 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:56:12.673998 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.68:0
--local_ip_for_outbound_sockets=127.5.139.68
--webserver_interface=127.5.139.68
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--builtin_ntp_servers=127.5.139.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:12.677966 8193 heartbeater.cc:347] Connected to a master server at 127.5.139.126:44591
I20240501 19:56:12.678171 8193 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:12.678658 8193 heartbeater.cc:511] Master 127.5.139.126:44591 requested a full tablet report, sending...
I20240501 19:56:12.679607 7743 ts_manager.cc:194] Registered new tserver with Master: 3905a8bbb6be48b3912362133e2c6aca (127.5.139.67:41301)
I20240501 19:56:12.680130 7743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.67:53433
W20240501 19:56:12.777320 8196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:12.777518 8196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:12.777586 8196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:12.780525 8196 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:12.780594 8196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:12.780694 8196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.68
I20240501 19:56:12.784524 8196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.139.68
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--never_fsync=true
--heap_profile_path=/tmp/kudu.8196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.68
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:12.785318 8196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:12.786336 8196 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:12.794461 8204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.795073 8201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.795740 8202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:12.796447 8196 server_base.cc:1034] running on GCE node
I20240501 19:56:12.796801 8196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:12.797346 8196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:12.798492 8196 hybrid_clock.cc:648] HybridClock initialized: now 1714593372798481 us; error 27 us; skew 500 ppm
I20240501 19:56:12.800235 8196 webserver.cc:460] Webserver started at http://127.5.139.68:36981/ using document root <none> and password file <none>
I20240501 19:56:12.800741 8196 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:12.800805 8196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:12.800993 8196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:12.802531 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7990904b9da04b0ea636ea8b71d77977"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.803042 8196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7990904b9da04b0ea636ea8b71d77977"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.806383 8196 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:12.808374 8210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.809309 8196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:12.809408 8196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7990904b9da04b0ea636ea8b71d77977"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.809490 8196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:12.827785 8196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:12.828354 8196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:12.828500 8196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:12.828963 8196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:12.829792 8196 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:12.829836 8196 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.829874 8196 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:12.829910 8196 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.837515 8196 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.68:40953
I20240501 19:56:12.837556 8323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.68:40953 every 8 connection(s)
I20240501 19:56:12.838335 8196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:12.839267 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 8196
I20240501 19:56:12.839357 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:56:12.841459 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.69:0
--local_ip_for_outbound_sockets=127.5.139.69
--webserver_interface=127.5.139.69
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--builtin_ntp_servers=127.5.139.84:33675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:12.847292 8324 heartbeater.cc:347] Connected to a master server at 127.5.139.126:44591
I20240501 19:56:12.847487 8324 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:12.847967 8324 heartbeater.cc:511] Master 127.5.139.126:44591 requested a full tablet report, sending...
I20240501 19:56:12.848937 7743 ts_manager.cc:194] Registered new tserver with Master: 7990904b9da04b0ea636ea8b71d77977 (127.5.139.68:40953)
I20240501 19:56:12.849731 7743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.68:48761
W20240501 19:56:12.947185 8327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:12.947388 8327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:12.947430 8327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:12.950505 8327 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:12.950569 8327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:12.950666 8327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.69
I20240501 19:56:12.954321 8327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:33675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.139.69
--webserver_port=0
--tserver_master_addrs=127.5.139.126:44591
--never_fsync=true
--heap_profile_path=/tmp/kudu.8327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.69
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:12.955081 8327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:12.956097 8327 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:12.964197 8332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.964834 8335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:12.965521 8333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:12.966241 8327 server_base.cc:1034] running on GCE node
I20240501 19:56:12.966572 8327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:12.967057 8327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:12.968214 8327 hybrid_clock.cc:648] HybridClock initialized: now 1714593372968199 us; error 28 us; skew 500 ppm
I20240501 19:56:12.969982 8327 webserver.cc:460] Webserver started at http://127.5.139.69:37945/ using document root <none> and password file <none>
I20240501 19:56:12.970495 8327 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:12.970554 8327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:12.970716 8327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:12.972239 8327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "19543f2643b949ba858029f745050048"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.972677 8327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "19543f2643b949ba858029f745050048"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.975725 8327 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:12.977697 8341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:12.978636 8327 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:12.978732 8327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "19543f2643b949ba858029f745050048"
format_stamp: "Formatted at 2024-05-01 19:56:12 on dist-test-slave-twc7"
I20240501 19:56:12.978812 8327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:13.004979 8327 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:13.005621 8327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:13.005785 8327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:13.006287 8327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:13.007313 8327 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:13.007364 8327 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.007424 8327 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:13.007457 8327 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:13.015714 8327 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.69:39677
I20240501 19:56:13.015749 8454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.69:39677 every 8 connection(s)
I20240501 19:56:13.016592 8327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:13.017009 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 8327
I20240501 19:56:13.017138 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:56:13.025379 8455 heartbeater.cc:347] Connected to a master server at 127.5.139.126:44591
I20240501 19:56:13.025596 8455 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:13.026154 8455 heartbeater.cc:511] Master 127.5.139.126:44591 requested a full tablet report, sending...
I20240501 19:56:13.027097 7743 ts_manager.cc:194] Registered new tserver with Master: 19543f2643b949ba858029f745050048 (127.5.139.69:39677)
I20240501 19:56:13.027688 7743 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.69:53875
I20240501 19:56:13.031025 5677 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:56:13.040550 7743 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38830:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:56:13.059072 8389 tablet_service.cc:1467] Processing CreateTablet for tablet 584ebb859709486082dbd557bcdd4a00 (DEFAULT_TABLE table=TestTable [id=bba40f4020544854bc5d4f826ff64156]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:13.059072 8127 tablet_service.cc:1467] Processing CreateTablet for tablet 584ebb859709486082dbd557bcdd4a00 (DEFAULT_TABLE table=TestTable [id=bba40f4020544854bc5d4f826ff64156]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:13.060060 8127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584ebb859709486082dbd557bcdd4a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:13.060060 8389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584ebb859709486082dbd557bcdd4a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:13.061393 8258 tablet_service.cc:1467] Processing CreateTablet for tablet 584ebb859709486082dbd557bcdd4a00 (DEFAULT_TABLE table=TestTable [id=bba40f4020544854bc5d4f826ff64156]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:13.062220 8258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 584ebb859709486082dbd557bcdd4a00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:13.066568 8473 tablet_bootstrap.cc:492] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca: Bootstrap starting.
I20240501 19:56:13.067200 8474 tablet_bootstrap.cc:492] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977: Bootstrap starting.
I20240501 19:56:13.068688 8473 tablet_bootstrap.cc:654] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:13.069028 8472 tablet_bootstrap.cc:492] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048: Bootstrap starting.
I20240501 19:56:13.069087 8474 tablet_bootstrap.cc:654] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:13.069552 8473 log.cc:824] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:13.069756 8474 log.cc:824] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:13.070839 8472 tablet_bootstrap.cc:654] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:13.071137 8473 tablet_bootstrap.cc:492] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca: No bootstrap required, opened a new log
I20240501 19:56:13.071296 8473 ts_tablet_manager.cc:1367] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:13.071439 8472 log.cc:824] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:13.072748 8474 tablet_bootstrap.cc:492] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977: No bootstrap required, opened a new log
I20240501 19:56:13.072901 8474 ts_tablet_manager.cc:1367] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 19:56:13.073005 8472 tablet_bootstrap.cc:492] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048: No bootstrap required, opened a new log
I20240501 19:56:13.073223 8472 ts_tablet_manager.cc:1367] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:13.074841 8473 raft_consensus.cc:348] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } }
I20240501 19:56:13.075058 8473 raft_consensus.cc:374] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:13.075126 8473 raft_consensus.cc:729] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3905a8bbb6be48b3912362133e2c6aca, State: Initialized, Role: FOLLOWER
I20240501 19:56:13.075677 8473 consensus_queue.cc:260] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } }
I20240501 19:56:13.075695 8474 raft_consensus.cc:348] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } }
I20240501 19:56:13.075891 8474 raft_consensus.cc:374] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:13.075946 8474 raft_consensus.cc:729] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7990904b9da04b0ea636ea8b71d77977, State: Initialized, Role: FOLLOWER
I20240501 19:56:13.076346 8474 consensus_queue.cc:260] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } }
I20240501 19:56:13.076503 8473 ts_tablet_manager.cc:1398] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240501 19:56:13.076748 8472 raft_consensus.cc:348] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } }
I20240501 19:56:13.076874 8193 heartbeater.cc:503] Master 127.5.139.126:44591 was elected leader, sending a full tablet report...
I20240501 19:56:13.076908 8472 raft_consensus.cc:374] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:13.076943 8472 raft_consensus.cc:729] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19543f2643b949ba858029f745050048, State: Initialized, Role: FOLLOWER
I20240501 19:56:13.077097 8474 ts_tablet_manager.cc:1398] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:13.077297 8472 consensus_queue.cc:260] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } }
I20240501 19:56:13.077389 8324 heartbeater.cc:503] Master 127.5.139.126:44591 was elected leader, sending a full tablet report...
I20240501 19:56:13.077891 8455 heartbeater.cc:503] Master 127.5.139.126:44591 was elected leader, sending a full tablet report...
I20240501 19:56:13.078035 8472 ts_tablet_manager.cc:1398] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20240501 19:56:13.089087 8325 tablet.cc:2369] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:13.108523 8479 raft_consensus.cc:482] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:13.108714 8479 raft_consensus.cc:504] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } }
I20240501 19:56:13.109743 8479 leader_election.cc:288] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19543f2643b949ba858029f745050048 (127.5.139.69:39677), 3905a8bbb6be48b3912362133e2c6aca (127.5.139.67:41301)
I20240501 19:56:13.112993 8409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584ebb859709486082dbd557bcdd4a00" candidate_uuid: "7990904b9da04b0ea636ea8b71d77977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19543f2643b949ba858029f745050048" is_pre_election: true
I20240501 19:56:13.113292 8409 raft_consensus.cc:2454] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7990904b9da04b0ea636ea8b71d77977 in term 0.
I20240501 19:56:13.113554 8147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584ebb859709486082dbd557bcdd4a00" candidate_uuid: "7990904b9da04b0ea636ea8b71d77977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3905a8bbb6be48b3912362133e2c6aca" is_pre_election: true
I20240501 19:56:13.113824 8147 raft_consensus.cc:2454] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7990904b9da04b0ea636ea8b71d77977 in term 0.
I20240501 19:56:13.113828 8213 leader_election.cc:302] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19543f2643b949ba858029f745050048, 7990904b9da04b0ea636ea8b71d77977; no voters:
I20240501 19:56:13.114049 8479 raft_consensus.cc:2781] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:13.114138 8479 raft_consensus.cc:482] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:13.114188 8479 raft_consensus.cc:3037] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:13.115012 8479 raft_consensus.cc:504] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } }
I20240501 19:56:13.115306 8479 leader_election.cc:288] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [CANDIDATE]: Term 1 election: Requested vote from peers 19543f2643b949ba858029f745050048 (127.5.139.69:39677), 3905a8bbb6be48b3912362133e2c6aca (127.5.139.67:41301)
I20240501 19:56:13.115658 8409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584ebb859709486082dbd557bcdd4a00" candidate_uuid: "7990904b9da04b0ea636ea8b71d77977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19543f2643b949ba858029f745050048"
I20240501 19:56:13.115686 8147 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "584ebb859709486082dbd557bcdd4a00" candidate_uuid: "7990904b9da04b0ea636ea8b71d77977" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3905a8bbb6be48b3912362133e2c6aca"
I20240501 19:56:13.115797 8409 raft_consensus.cc:3037] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:13.115797 8147 raft_consensus.cc:3037] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:13.116868 8147 raft_consensus.cc:2454] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7990904b9da04b0ea636ea8b71d77977 in term 1.
I20240501 19:56:13.116868 8409 raft_consensus.cc:2454] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7990904b9da04b0ea636ea8b71d77977 in term 1.
I20240501 19:56:13.117192 8213 leader_election.cc:302] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3905a8bbb6be48b3912362133e2c6aca, 7990904b9da04b0ea636ea8b71d77977; no voters:
I20240501 19:56:13.117358 8479 raft_consensus.cc:2781] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:13.117561 8479 raft_consensus.cc:686] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [term 1 LEADER]: Becoming Leader. State: Replica: 7990904b9da04b0ea636ea8b71d77977, State: Running, Role: LEADER
I20240501 19:56:13.117826 8479 consensus_queue.cc:237] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } }
I20240501 19:56:13.120460 7738 catalog_manager.cc:5484] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7990904b9da04b0ea636ea8b71d77977 (127.5.139.68). New cstate: current_term: 1 leader_uuid: "7990904b9da04b0ea636ea8b71d77977" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7990904b9da04b0ea636ea8b71d77977" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 40953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:13.132388 5677 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 19:56:13.133867 5677 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 9682d399a91e4df8ade08ee2f96692b8 to finish bootstrapping
I20240501 19:56:13.139525 5677 ts_itest-base.cc:233] Waiting for 0 tablets on tserver cbdb01e549004ac0a811cb76dc55527d to finish bootstrapping
I20240501 19:56:13.144836 5677 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 3905a8bbb6be48b3912362133e2c6aca to finish bootstrapping
I20240501 19:56:13.149991 5677 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7990904b9da04b0ea636ea8b71d77977 to finish bootstrapping
I20240501 19:56:13.154747 5677 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 19543f2643b949ba858029f745050048 to finish bootstrapping
W20240501 19:56:13.173326 8194 tablet.cc:2369] T 584ebb859709486082dbd557bcdd4a00 P 3905a8bbb6be48b3912362133e2c6aca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:13.267480 8456 tablet.cc:2369] T 584ebb859709486082dbd557bcdd4a00 P 19543f2643b949ba858029f745050048: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:13.303583 7931 heartbeater.cc:503] Master 127.5.139.126:44591 was elected leader, sending a full tablet report...
I20240501 19:56:13.336804 8389 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.336892 8127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.336969 7865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.337399 7996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.339781 8258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:13.423044 8389 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.423466 8258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:13.423862 8127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.424803 7865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.425407 7996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.492872 8062 heartbeater.cc:503] Master 127.5.139.126:44591 was elected leader, sending a full tablet report...
I20240501 19:56:13.591717 8479 consensus_queue.cc:1035] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3905a8bbb6be48b3912362133e2c6aca" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 41301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:13.596741 8567 consensus_queue.cc:1035] T 584ebb859709486082dbd557bcdd4a00 P 7990904b9da04b0ea636ea8b71d77977 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19543f2643b949ba858029f745050048" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 39677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:13.634282 7865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.634727 8258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:13.635107 8389 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.635274 8127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.636451 7996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.720728 8389 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.720981 7865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.722241 8258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:13.723413 7996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.729193 8127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.823180 8127 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.823428 8389 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.823475 7865 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.824023 8258 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:13.825186 7996 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:13.867472 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 7803
I20240501 19:56:13.873243 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 7934
I20240501 19:56:13.878854 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 8065
I20240501 19:56:13.884464 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 8196
I20240501 19:56:13.890201 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 8327
I20240501 19:56:13.895748 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 7712
2024-05-01T19:56:13Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (2066 ms)
[----------] 1 test from RebalanceStartCriteriaTest (2066 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240501 19:56:13.917716 5677 test_util.cc:268] Using random seed: -520318448
I20240501 19:56:13.918426 5677 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:56:13.918460 5677 ts_itest-base.cc:113] --------------
I20240501 19:56:13.918480 5677 ts_itest-base.cc:114] 7 tablet servers
I20240501 19:56:13.918498 5677 ts_itest-base.cc:115] 2 replicas per TS
I20240501 19:56:13.918524 5677 ts_itest-base.cc:116] --------------
2024-05-01T19:56:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:13Z Disabled control of system clock
I20240501 19:56:13.926584 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.139.126:34427
--webserver_interface=127.5.139.126
--webserver_port=0
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.139.126:34427
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 19:56:14.026790 8666 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 19:56:14.027031 8666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:14.027058 8666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:14.027130 8666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:14.030122 8666 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:14.030197 8666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:14.030222 8666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:14.030261 8666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:14.030294 8666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:14.033915 8666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.139.126:34427
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.139.126:34427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.139.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.8666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:14.034834 8666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:14.035951 8666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:14.043756 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.044361 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.045058 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.045789 8666 server_base.cc:1034] running on GCE node
I20240501 19:56:14.046294 8666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:14.047230 8666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:14.048393 8666 hybrid_clock.cc:648] HybridClock initialized: now 1714593374048369 us; error 40 us; skew 500 ppm
I20240501 19:56:14.050374 8666 webserver.cc:460] Webserver started at http://127.5.139.126:39411/ using document root <none> and password file <none>
I20240501 19:56:14.050910 8666 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:14.050966 8666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:14.051142 8666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:14.052744 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2ee3823193724caab53a0a77e70e18df"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.053228 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2ee3823193724caab53a0a77e70e18df"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.056419 8666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 19:56:14.058616 8680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.059574 8666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:14.059700 8666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2ee3823193724caab53a0a77e70e18df"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.059787 8666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:14.078472 8666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:14.079125 8666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:14.079289 8666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:14.086158 8666 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.126:34427
I20240501 19:56:14.086169 8732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.126:34427 every 8 connection(s)
I20240501 19:56:14.087081 8666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:56:14.089452 8733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:14.091395 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 8666
I20240501 19:56:14.091531 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:56:14.095000 8733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df: Bootstrap starting.
I20240501 19:56:14.097239 8733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:14.098006 8733 log.cc:824] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:14.099800 8733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df: No bootstrap required, opened a new log
I20240501 19:56:14.102270 8733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee3823193724caab53a0a77e70e18df" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 34427 } }
I20240501 19:56:14.102432 8733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:14.102474 8733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ee3823193724caab53a0a77e70e18df, State: Initialized, Role: FOLLOWER
I20240501 19:56:14.102821 8733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee3823193724caab53a0a77e70e18df" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 34427 } }
I20240501 19:56:14.102931 8733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:14.102977 8733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:14.103056 8733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:14.103708 8733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee3823193724caab53a0a77e70e18df" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 34427 } }
I20240501 19:56:14.103976 8733 leader_election.cc:302] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ee3823193724caab53a0a77e70e18df; no voters:
I20240501 19:56:14.104192 8733 leader_election.cc:288] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:14.104537 8738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:14.104758 8738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [term 1 LEADER]: Becoming Leader. State: Replica: 2ee3823193724caab53a0a77e70e18df, State: Running, Role: LEADER
I20240501 19:56:14.105069 8738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee3823193724caab53a0a77e70e18df" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 34427 } }
I20240501 19:56:14.105237 8733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:14.106420 8740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ee3823193724caab53a0a77e70e18df. Latest consensus state: current_term: 1 leader_uuid: "2ee3823193724caab53a0a77e70e18df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee3823193724caab53a0a77e70e18df" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 34427 } } }
I20240501 19:56:14.106516 8740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:14.106838 8739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ee3823193724caab53a0a77e70e18df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ee3823193724caab53a0a77e70e18df" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 34427 } } }
I20240501 19:56:14.106935 8739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df [sys.catalog]: This master's current role is: LEADER
W20240501 19:56:14.109731 8753 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:14.109802 8753 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:14.109874 8748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:14.110564 8748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:14.115113 8748 catalog_manager.cc:1347] Generated new cluster ID: f19f5660c9894d8cb18968321dec9316
I20240501 19:56:14.115173 8748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:14.128293 8748 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:14.129153 8748 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:14.136906 8748 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 2ee3823193724caab53a0a77e70e18df: Generated new TSK 0
I20240501 19:56:14.137529 8748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:14.144970 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.65:0
--local_ip_for_outbound_sockets=127.5.139.65
--webserver_interface=127.5.139.65
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:14.248468 8757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:14.248674 8757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:14.248719 8757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:14.251796 8757 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:14.251859 8757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:14.251947 8757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.65
I20240501 19:56:14.255488 8757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.139.65
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.8757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.65
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:14.256304 8757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:14.257411 8757 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:14.265615 8765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.266387 8762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.266965 8763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.267724 8757 server_base.cc:1034] running on GCE node
I20240501 19:56:14.268057 8757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:14.268553 8757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:14.269701 8757 hybrid_clock.cc:648] HybridClock initialized: now 1714593374269689 us; error 25 us; skew 500 ppm
I20240501 19:56:14.274864 8757 webserver.cc:460] Webserver started at http://127.5.139.65:34137/ using document root <none> and password file <none>
I20240501 19:56:14.275388 8757 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:14.275450 8757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:14.275617 8757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:14.277227 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "0f2b49c6d3dc49fabee40802e73ac982"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.277666 8757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "0f2b49c6d3dc49fabee40802e73ac982"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.280710 8757 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:14.282780 8771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.283730 8757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:14.283830 8757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "0f2b49c6d3dc49fabee40802e73ac982"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.283915 8757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:14.308815 8757 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:14.309407 8757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:14.309558 8757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:14.310072 8757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:14.310910 8757 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:14.310971 8757 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.311026 8757 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:14.311057 8757 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.319473 8757 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.65:38731
I20240501 19:56:14.319515 8884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.65:38731 every 8 connection(s)
I20240501 19:56:14.320370 8757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:56:14.328366 8885 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:14.328573 8885 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:14.329118 8885 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:14.330579 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 8757
I20240501 19:56:14.330561 8697 ts_manager.cc:194] Registered new tserver with Master: 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:14.330705 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:56:14.331936 8697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.65:35195
I20240501 19:56:14.332901 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.66:0
--local_ip_for_outbound_sockets=127.5.139.66
--webserver_interface=127.5.139.66
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:14.437692 8888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:14.437902 8888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:14.437942 8888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:14.440896 8888 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:14.440960 8888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:14.441084 8888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.66
I20240501 19:56:14.444538 8888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.139.66
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.8888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.66
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:14.445362 8888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:14.446399 8888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:14.454414 8894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.455104 8893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.455775 8896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.456529 8888 server_base.cc:1034] running on GCE node
I20240501 19:56:14.456879 8888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:14.457415 8888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:14.458575 8888 hybrid_clock.cc:648] HybridClock initialized: now 1714593374458564 us; error 22 us; skew 500 ppm
I20240501 19:56:14.460280 8888 webserver.cc:460] Webserver started at http://127.5.139.66:36299/ using document root <none> and password file <none>
I20240501 19:56:14.460791 8888 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:14.460850 8888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:14.461011 8888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:14.462711 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ac13bdb319c448dba072fb9e62d6893b"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.463176 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ac13bdb319c448dba072fb9e62d6893b"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.466333 8888 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20240501 19:56:14.468340 8902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.469333 8888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.002s
I20240501 19:56:14.469437 8888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ac13bdb319c448dba072fb9e62d6893b"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.469516 8888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:14.479701 8888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:14.480230 8888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:14.480387 8888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:14.480859 8888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:14.481909 8888 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:14.481976 8888 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.482040 8888 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:14.482069 8888 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.491135 8888 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.66:43725
I20240501 19:56:14.491158 9015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.66:43725 every 8 connection(s)
I20240501 19:56:14.491997 8888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:56:14.498188 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 8888
I20240501 19:56:14.498304 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:56:14.500417 9016 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:14.500522 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.67:0
--local_ip_for_outbound_sockets=127.5.139.67
--webserver_interface=127.5.139.67
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:14.500627 9016 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:14.501164 9016 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:14.502144 8697 ts_manager.cc:194] Registered new tserver with Master: ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:14.502661 8697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.66:35599
W20240501 19:56:14.606184 9019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:14.606421 9019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:14.606477 9019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:14.609808 9019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:14.609880 9019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:14.609977 9019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.67
I20240501 19:56:14.613646 9019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.139.67
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.9019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.67
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:14.614564 9019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:14.615674 9019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:14.624239 9024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.624918 9025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.625650 9027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.626322 9019 server_base.cc:1034] running on GCE node
I20240501 19:56:14.626677 9019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:14.627255 9019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:14.628417 9019 hybrid_clock.cc:648] HybridClock initialized: now 1714593374628404 us; error 29 us; skew 500 ppm
I20240501 19:56:14.630265 9019 webserver.cc:460] Webserver started at http://127.5.139.67:40049/ using document root <none> and password file <none>
I20240501 19:56:14.630811 9019 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:14.630872 9019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:14.631044 9019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:14.632620 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.633141 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.636216 9019 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:14.638358 9033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.639271 9019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:14.639369 9019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.639446 9019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:14.662187 9019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:14.662920 9019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:14.663097 9019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:14.663653 9019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:14.664563 9019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:14.664613 9019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.664675 9019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:14.664708 9019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.673022 9019 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.67:35389
I20240501 19:56:14.673099 9146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.67:35389 every 8 connection(s)
I20240501 19:56:14.674001 9019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:56:14.676862 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 9019
I20240501 19:56:14.676976 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:56:14.679199 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.68:0
--local_ip_for_outbound_sockets=127.5.139.68
--webserver_interface=127.5.139.68
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:14.683007 9147 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:14.683200 9147 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:14.683681 9147 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:14.684659 8697 ts_manager.cc:194] Registered new tserver with Master: b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:14.685482 8697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.67:45967
W20240501 19:56:14.780484 9150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:14.780659 9150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:14.780699 9150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:14.783562 9150 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:14.783628 9150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:14.783692 9150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.68
I20240501 19:56:14.787087 9150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.139.68
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.9150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.68
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:14.787839 9150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:14.788865 9150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:14.796648 9158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.797281 9155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.797946 9156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.798671 9150 server_base.cc:1034] running on GCE node
I20240501 19:56:14.798975 9150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:14.799456 9150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:14.800628 9150 hybrid_clock.cc:648] HybridClock initialized: now 1714593374800616 us; error 24 us; skew 500 ppm
I20240501 19:56:14.802263 9150 webserver.cc:460] Webserver started at http://127.5.139.68:37379/ using document root <none> and password file <none>
I20240501 19:56:14.802767 9150 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:14.802824 9150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:14.802981 9150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:14.804446 9150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c9564f3b39b549c1bd995354fa135c5f"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.804885 9150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c9564f3b39b549c1bd995354fa135c5f"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.808107 9150 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:14.810104 9164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.811017 9150 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:14.811111 9150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c9564f3b39b549c1bd995354fa135c5f"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.811188 9150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:14.829182 9150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:14.829708 9150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:14.829847 9150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:14.830302 9150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:14.831079 9150 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:14.831126 9150 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.831183 9150 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:14.831212 9150 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.839274 9150 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.68:46413
I20240501 19:56:14.839298 9277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.68:46413 every 8 connection(s)
I20240501 19:56:14.840095 9150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:14.844806 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 9150
I20240501 19:56:14.844916 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:56:14.846884 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.69:0
--local_ip_for_outbound_sockets=127.5.139.69
--webserver_interface=127.5.139.69
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:14.849094 9278 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:14.849294 9278 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:14.849758 9278 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:14.850695 8697 ts_manager.cc:194] Registered new tserver with Master: c9564f3b39b549c1bd995354fa135c5f (127.5.139.68:46413)
I20240501 19:56:14.851218 8697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.68:56063
W20240501 19:56:14.948789 9281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:14.948971 9281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:14.949070 9281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:14.951965 9281 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:14.952034 9281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:14.952123 9281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.69
I20240501 19:56:14.955538 9281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.139.69
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.9281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.69
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:14.956259 9281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:14.957271 9281 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:14.965107 9287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.965899 9286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:14.966845 9289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:14.967545 9281 server_base.cc:1034] running on GCE node
I20240501 19:56:14.967895 9281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:14.968397 9281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:14.969547 9281 hybrid_clock.cc:648] HybridClock initialized: now 1714593374969536 us; error 23 us; skew 500 ppm
I20240501 19:56:14.971287 9281 webserver.cc:460] Webserver started at http://127.5.139.69:42907/ using document root <none> and password file <none>
I20240501 19:56:14.971829 9281 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:14.971889 9281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:14.972075 9281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:14.973773 9281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.974316 9281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.977450 9281 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20240501 19:56:14.979511 9295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:14.980438 9281 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:14.980548 9281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:14.980625 9281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:15.011941 9281 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:15.012550 9281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:15.012717 9281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:15.013252 9281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:15.014156 9281 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:15.014211 9281 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.014269 9281 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:15.014297 9281 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.022805 9281 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.69:42483
I20240501 19:56:15.022830 9408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.69:42483 every 8 connection(s)
I20240501 19:56:15.023689 9281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:15.031792 9409 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:15.031976 9409 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:15.032461 9409 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:15.032601 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 9281
I20240501 19:56:15.032723 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:56:15.033565 8697 ts_manager.cc:194] Registered new tserver with Master: 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483)
I20240501 19:56:15.034173 8697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.69:40655
I20240501 19:56:15.034768 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.70:0
--local_ip_for_outbound_sockets=127.5.139.70
--webserver_interface=127.5.139.70
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:15.140125 9412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:15.140300 9412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:15.140342 9412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:15.143321 9412 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:15.143378 9412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:15.143468 9412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.70
I20240501 19:56:15.146926 9412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.139.70
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.9412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.70
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:15.147626 9412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:15.148578 9412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:15.156514 9420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.157248 9417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.157830 9418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:15.158557 9412 server_base.cc:1034] running on GCE node
I20240501 19:56:15.158841 9412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:15.159307 9412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:15.160482 9412 hybrid_clock.cc:648] HybridClock initialized: now 1714593375160462 us; error 35 us; skew 500 ppm
I20240501 19:56:15.162178 9412 webserver.cc:460] Webserver started at http://127.5.139.70:40021/ using document root <none> and password file <none>
I20240501 19:56:15.162667 9412 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:15.162725 9412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:15.162906 9412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:15.164433 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "6b4f42453a864314adc86aba2c3b3556"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-twc7"
I20240501 19:56:15.164891 9412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "6b4f42453a864314adc86aba2c3b3556"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-twc7"
I20240501 19:56:15.167804 9412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:15.169816 9426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.170696 9412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:15.170783 9412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6b4f42453a864314adc86aba2c3b3556"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-twc7"
I20240501 19:56:15.170862 9412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:15.188362 9412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:15.188861 9412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:15.189003 9412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:15.189478 9412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:15.190276 9412 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:15.190321 9412 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.190361 9412 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:15.190390 9412 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.198057 9412 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.70:43441
I20240501 19:56:15.198117 9539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.70:43441 every 8 connection(s)
I20240501 19:56:15.198858 9412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:15.200042 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 9412
I20240501 19:56:15.200156 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:56:15.202287 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.71:0
--local_ip_for_outbound_sockets=127.5.139.71
--webserver_interface=127.5.139.71
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:15.207561 9540 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:15.207751 9540 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:15.208199 9540 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:15.209071 8697 ts_manager.cc:194] Registered new tserver with Master: 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441)
I20240501 19:56:15.209681 8697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.70:36669
W20240501 19:56:15.303704 9543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:15.303880 9543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:15.303949 9543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:15.306831 9543 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:15.306896 9543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:15.306978 9543 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.71
I20240501 19:56:15.310369 9543 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.139.71
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.9543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.71
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:15.311060 9543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:15.312023 9543 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:15.319881 9551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.320616 9548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:15.321341 9549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:15.321949 9543 server_base.cc:1034] running on GCE node
I20240501 19:56:15.322233 9543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:15.322708 9543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:15.323864 9543 hybrid_clock.cc:648] HybridClock initialized: now 1714593375323850 us; error 29 us; skew 500 ppm
I20240501 19:56:15.325520 9543 webserver.cc:460] Webserver started at http://127.5.139.71:34883/ using document root <none> and password file <none>
I20240501 19:56:15.326010 9543 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:15.326081 9543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:15.326237 9543 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:15.327699 9543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "14798909ff724444a8be03c45c41929a"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-twc7"
I20240501 19:56:15.328135 9543 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "14798909ff724444a8be03c45c41929a"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-twc7"
I20240501 19:56:15.331095 9543 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20240501 19:56:15.333128 9557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.334013 8885 heartbeater.cc:503] Master 127.5.139.126:34427 was elected leader, sending a full tablet report...
I20240501 19:56:15.334038 9543 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:15.334126 9543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "14798909ff724444a8be03c45c41929a"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-twc7"
I20240501 19:56:15.334208 9543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:15.349496 9543 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:15.350123 9543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:15.350361 9543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:15.350955 9543 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:15.351815 9543 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:15.351864 9543 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.351931 9543 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:15.351959 9543 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:15.360266 9543 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.71:38393
I20240501 19:56:15.360277 9670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.71:38393 every 8 connection(s)
I20240501 19:56:15.361220 9543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:15.367839 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 9543
I20240501 19:56:15.367955 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:56:15.370028 9671 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:15.370232 9671 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:15.370743 9671 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:15.371624 8697 ts_manager.cc:194] Registered new tserver with Master: 14798909ff724444a8be03c45c41929a (127.5.139.71:38393)
I20240501 19:56:15.372162 8697 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.71:57213
I20240501 19:56:15.381938 5677 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 19:56:15.390336 8697 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40084:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 19:56:15.405966 8819 tablet_service.cc:1467] Processing CreateTablet for tablet c6e64c2bfdce4caf9c91618400c7f5d6 (DEFAULT_TABLE table=TestTable [id=667da58c9fb24a419759f30314644e23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:15.406198 9605 tablet_service.cc:1467] Processing CreateTablet for tablet c6e64c2bfdce4caf9c91618400c7f5d6 (DEFAULT_TABLE table=TestTable [id=667da58c9fb24a419759f30314644e23]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:15.406849 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e64c2bfdce4caf9c91618400c7f5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:15.406960 9605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e64c2bfdce4caf9c91618400c7f5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:15.411401 9687 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:15.411451 9688 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Bootstrap starting.
I20240501 19:56:15.413404 9688 tablet_bootstrap.cc:654] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:15.413404 9687 tablet_bootstrap.cc:654] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:15.414101 9687 log.cc:824] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:15.414101 9688 log.cc:824] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:15.415484 9687 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:15.415498 9688 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: No bootstrap required, opened a new log
I20240501 19:56:15.415635 9687 ts_tablet_manager.cc:1367] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:15.415634 9688 ts_tablet_manager.cc:1367] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:15.418568 9687 raft_consensus.cc:348] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:15.418577 9688 raft_consensus.cc:348] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:15.418735 9687 raft_consensus.cc:374] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:15.418740 9688 raft_consensus.cc:374] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:15.418776 9687 raft_consensus.cc:729] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:15.418792 9688 raft_consensus.cc:729] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14798909ff724444a8be03c45c41929a, State: Initialized, Role: FOLLOWER
I20240501 19:56:15.419164 9687 consensus_queue.cc:260] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:15.419165 9688 consensus_queue.cc:260] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:15.419894 9687 ts_tablet_manager.cc:1398] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:15.419932 9688 ts_tablet_manager.cc:1398] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:15.420090 9671 heartbeater.cc:503] Master 127.5.139.126:34427 was elected leader, sending a full tablet report...
I20240501 19:56:15.491278 9692 raft_consensus.cc:482] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:15.491462 9692 raft_consensus.cc:504] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:15.492420 9692 leader_election.cc:288] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:15.495965 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e64c2bfdce4caf9c91618400c7f5d6" candidate_uuid: "14798909ff724444a8be03c45c41929a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982" is_pre_election: true
I20240501 19:56:15.496235 8839 raft_consensus.cc:2454] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14798909ff724444a8be03c45c41929a in term 0.
I20240501 19:56:15.496676 9558 leader_election.cc:302] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, 14798909ff724444a8be03c45c41929a; no voters:
I20240501 19:56:15.496903 9692 raft_consensus.cc:2781] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:15.496971 9692 raft_consensus.cc:482] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:15.497001 9692 raft_consensus.cc:3037] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:15.497834 9692 raft_consensus.cc:504] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:15.498091 9692 leader_election.cc:288] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 1 election: Requested vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:15.498415 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e64c2bfdce4caf9c91618400c7f5d6" candidate_uuid: "14798909ff724444a8be03c45c41929a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
I20240501 19:56:15.498520 8839 raft_consensus.cc:3037] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:15.499403 8839 raft_consensus.cc:2454] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14798909ff724444a8be03c45c41929a in term 1.
I20240501 19:56:15.499684 9558 leader_election.cc:302] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, 14798909ff724444a8be03c45c41929a; no voters:
I20240501 19:56:15.499858 9692 raft_consensus.cc:2781] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:15.500074 9692 raft_consensus.cc:686] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 LEADER]: Becoming Leader. State: Replica: 14798909ff724444a8be03c45c41929a, State: Running, Role: LEADER
I20240501 19:56:15.500350 9692 consensus_queue.cc:237] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:15.503021 8695 catalog_manager.cc:5484] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a reported cstate change: term changed from 0 to 1, leader changed from <none> to 14798909ff724444a8be03c45c41929a (127.5.139.71). New cstate: current_term: 1 leader_uuid: "14798909ff724444a8be03c45c41929a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:15.504406 9016 heartbeater.cc:503] Master 127.5.139.126:34427 was elected leader, sending a full tablet report...
I20240501 19:56:15.527199 5677 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 19:56:15.528623 5677 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 0f2b49c6d3dc49fabee40802e73ac982 to finish bootstrapping
I20240501 19:56:15.534098 5677 ts_itest-base.cc:233] Waiting for 0 tablets on tserver ac13bdb319c448dba072fb9e62d6893b to finish bootstrapping
I20240501 19:56:15.539721 5677 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b3b45c9058d84b4e9ace42c9154ab0e0 to finish bootstrapping
I20240501 19:56:15.545063 5677 ts_itest-base.cc:233] Waiting for 0 tablets on tserver c9564f3b39b549c1bd995354fa135c5f to finish bootstrapping
I20240501 19:56:15.550124 5677 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6977f9a52e9b4d06bcb12f9bee61553b to finish bootstrapping
I20240501 19:56:15.554980 5677 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 6b4f42453a864314adc86aba2c3b3556 to finish bootstrapping
I20240501 19:56:15.559803 5677 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 14798909ff724444a8be03c45c41929a to finish bootstrapping
I20240501 19:56:15.564399 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 9150
I20240501 19:56:15.570484 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 9281
W20240501 19:56:15.571835 8886 tablet.cc:2369] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:15.575981 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 9412
I20240501 19:56:15.581220 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 9543
I20240501 19:56:15.687361 9147 heartbeater.cc:503] Master 127.5.139.126:34427 was elected leader, sending a full tablet report...
I20240501 19:56:17.740128 9701 raft_consensus.cc:482] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:17.740274 9701 raft_consensus.cc:504] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:17.741261 9701 leader_election.cc:288] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14798909ff724444a8be03c45c41929a (127.5.139.71:38393)
W20240501 19:56:17.742040 8773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:38393: connect: Connection refused (error 111)
W20240501 19:56:17.742879 8773 leader_election.cc:334] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 14798909ff724444a8be03c45c41929a (127.5.139.71:38393): Network error: Client connection negotiation failed: client connection to 127.5.139.71:38393: connect: Connection refused (error 111)
I20240501 19:56:17.742970 8773 leader_election.cc:302] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982; no voters: 14798909ff724444a8be03c45c41929a
I20240501 19:56:17.743191 9701 raft_consensus.cc:2726] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:19.247355 9704 raft_consensus.cc:482] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:19.247486 9704 raft_consensus.cc:504] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:19.247888 9704 leader_election.cc:288] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14798909ff724444a8be03c45c41929a (127.5.139.71:38393)
W20240501 19:56:19.249485 8773 leader_election.cc:334] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 14798909ff724444a8be03c45c41929a (127.5.139.71:38393): Network error: Client connection negotiation failed: client connection to 127.5.139.71:38393: connect: Connection refused (error 111)
I20240501 19:56:19.249568 8773 leader_election.cc:302] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982; no voters: 14798909ff724444a8be03c45c41929a
I20240501 19:56:19.249717 9704 raft_consensus.cc:2726] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:19.337908 8697 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40084:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:19.345242 8819 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4e313b03cc44d0a2f7864f7e9d4f90 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:19.345595 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4e313b03cc44d0a2f7864f7e9d4f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.347963 9715 tablet_bootstrap.cc:492] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:19.348863 9715 tablet_bootstrap.cc:654] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.350256 9715 tablet_bootstrap.cc:492] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:19.350368 9715 ts_tablet_manager.cc:1367] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:19.350912 9715 raft_consensus.cc:348] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.351042 9715 raft_consensus.cc:374] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.351094 9715 raft_consensus.cc:729] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.351289 9715 consensus_queue.cc:260] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.351074 9080 tablet_service.cc:1467] Processing CreateTablet for tablet 68c8311312bd4b50bdbedf414ada6bd2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:19.351083 8949 tablet_service.cc:1467] Processing CreateTablet for tablet 68c8311312bd4b50bdbedf414ada6bd2 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:19.351608 9715 ts_tablet_manager.cc:1398] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:19.351297 8950 tablet_service.cc:1467] Processing CreateTablet for tablet 6db903f127ce471d9cb87537a3cb7286 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:19.351089 8948 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4e313b03cc44d0a2f7864f7e9d4f90 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:19.352030 8949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c8311312bd4b50bdbedf414ada6bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.352031 9080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c8311312bd4b50bdbedf414ada6bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.352327 9081 tablet_service.cc:1467] Processing CreateTablet for tablet 6db903f127ce471d9cb87537a3cb7286 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:19.352564 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db903f127ce471d9cb87537a3cb7286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.353590 8950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db903f127ce471d9cb87537a3cb7286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.353771 8948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4e313b03cc44d0a2f7864f7e9d4f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:19.358433 9717 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:19.358722 9718 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:19.360210 9718 tablet_bootstrap.cc:654] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.360224 9717 tablet_bootstrap.cc:654] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.360803 9718 log.cc:824] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:19.360989 9717 log.cc:824] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:19.362362 9717 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:19.362362 9718 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:19.362527 9717 ts_tablet_manager.cc:1367] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 19:56:19.362527 9718 ts_tablet_manager.cc:1367] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 19:56:19.365793 9717 raft_consensus.cc:348] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.366006 9717 raft_consensus.cc:374] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.366063 9717 raft_consensus.cc:729] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.366182 9718 raft_consensus.cc:348] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.366349 9718 raft_consensus.cc:374] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.366400 9718 raft_consensus.cc:729] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.366492 9717 consensus_queue.cc:260] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.366775 9718 consensus_queue.cc:260] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.367307 9717 ts_tablet_manager.cc:1398] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240501 19:56:19.367617 9717 tablet_bootstrap.cc:492] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:19.368309 9718 ts_tablet_manager.cc:1398] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20240501 19:56:19.368577 9718 tablet_bootstrap.cc:492] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:19.369264 9717 tablet_bootstrap.cc:654] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.369611 9718 tablet_bootstrap.cc:654] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.370637 9717 tablet_bootstrap.cc:492] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:19.370713 9717 ts_tablet_manager.cc:1367] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:19.371146 9717 raft_consensus.cc:348] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.371229 9717 raft_consensus.cc:374] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.371263 9717 raft_consensus.cc:729] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.371384 9717 consensus_queue.cc:260] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.371579 9717 ts_tablet_manager.cc:1398] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:19.371739 9717 tablet_bootstrap.cc:492] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:19.372495 9717 tablet_bootstrap.cc:654] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:19.373351 9718 tablet_bootstrap.cc:492] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:19.373438 9718 ts_tablet_manager.cc:1367] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:56:19.373548 9717 tablet_bootstrap.cc:492] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:19.373623 9717 ts_tablet_manager.cc:1367] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:19.373975 9717 raft_consensus.cc:348] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.373960 9718 raft_consensus.cc:348] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.374048 9718 raft_consensus.cc:374] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.374048 9717 raft_consensus.cc:374] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:19.374080 9718 raft_consensus.cc:729] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.374080 9717 raft_consensus.cc:729] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:19.374197 9717 consensus_queue.cc:260] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.374197 9718 consensus_queue.cc:260] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.374442 9718 ts_tablet_manager.cc:1398] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:19.374455 9717 ts_tablet_manager.cc:1398] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240501 19:56:19.427995 9148 tablet.cc:2369] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:19.496182 9017 tablet.cc:2369] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:19.545624 9721 raft_consensus.cc:482] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:19.545823 9721 raft_consensus.cc:504] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.546865 9721 leader_election.cc:288] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:19.550760 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c8311312bd4b50bdbedf414ada6bd2" candidate_uuid: "ac13bdb319c448dba072fb9e62d6893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" is_pre_election: true
I20240501 19:56:19.551059 9101 raft_consensus.cc:2454] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac13bdb319c448dba072fb9e62d6893b in term 0.
I20240501 19:56:19.551506 8906 leader_election.cc:302] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:19.551786 9721 raft_consensus.cc:2781] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:19.551862 9721 raft_consensus.cc:482] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:19.551923 9721 raft_consensus.cc:3037] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.552845 9721 raft_consensus.cc:504] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.553198 9721 leader_election.cc:288] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 election: Requested vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:19.553614 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c8311312bd4b50bdbedf414ada6bd2" candidate_uuid: "ac13bdb319c448dba072fb9e62d6893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
I20240501 19:56:19.553743 9101 raft_consensus.cc:3037] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.554605 9101 raft_consensus.cc:2454] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac13bdb319c448dba072fb9e62d6893b in term 1.
I20240501 19:56:19.554915 8906 leader_election.cc:302] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:19.555078 9721 raft_consensus.cc:2781] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:19.555325 9721 raft_consensus.cc:686] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Becoming Leader. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Running, Role: LEADER
I20240501 19:56:19.555644 9721 consensus_queue.cc:237] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.558662 8697 catalog_manager.cc:5484] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac13bdb319c448dba072fb9e62d6893b (127.5.139.66). New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:19.599419 9721 raft_consensus.cc:482] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:19.599547 9721 raft_consensus.cc:504] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.599889 9721 leader_election.cc:288] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:19.600386 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db903f127ce471d9cb87537a3cb7286" candidate_uuid: "ac13bdb319c448dba072fb9e62d6893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" is_pre_election: true
I20240501 19:56:19.600512 9101 raft_consensus.cc:2454] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac13bdb319c448dba072fb9e62d6893b in term 0.
I20240501 19:56:19.600797 8906 leader_election.cc:302] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:19.600991 9721 raft_consensus.cc:2781] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:19.601061 9721 raft_consensus.cc:482] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:19.601120 9721 raft_consensus.cc:3037] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.601912 9721 raft_consensus.cc:504] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.602174 9721 leader_election.cc:288] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 election: Requested vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:19.602551 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db903f127ce471d9cb87537a3cb7286" candidate_uuid: "ac13bdb319c448dba072fb9e62d6893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
I20240501 19:56:19.602677 9101 raft_consensus.cc:3037] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.603385 9101 raft_consensus.cc:2454] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac13bdb319c448dba072fb9e62d6893b in term 1.
I20240501 19:56:19.603698 8906 leader_election.cc:302] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:19.603909 9721 raft_consensus.cc:2781] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:19.603991 9721 raft_consensus.cc:686] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Becoming Leader. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Running, Role: LEADER
I20240501 19:56:19.604137 9721 consensus_queue.cc:237] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.605643 8697 catalog_manager.cc:5484] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac13bdb319c448dba072fb9e62d6893b (127.5.139.66). New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:19.727870 9704 raft_consensus.cc:482] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:19.727991 9704 raft_consensus.cc:504] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.728415 9704 leader_election.cc:288] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:19.732616 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4e313b03cc44d0a2f7864f7e9d4f90" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b" is_pre_election: true
I20240501 19:56:19.732875 8970 raft_consensus.cc:2454] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 0.
I20240501 19:56:19.733390 8773 leader_election.cc:302] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:19.733628 9704 raft_consensus.cc:2781] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:19.733705 9704 raft_consensus.cc:482] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:19.733762 9704 raft_consensus.cc:3037] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.734587 9704 raft_consensus.cc:504] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.734853 9704 leader_election.cc:288] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Requested vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:19.735318 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4e313b03cc44d0a2f7864f7e9d4f90" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
I20240501 19:56:19.735459 8970 raft_consensus.cc:3037] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:19.736719 8970 raft_consensus.cc:2454] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 1.
I20240501 19:56:19.737092 8773 leader_election.cc:302] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:19.737294 9704 raft_consensus.cc:2781] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:19.737519 9704 raft_consensus.cc:686] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Running, Role: LEADER
I20240501 19:56:19.737847 9704 consensus_queue.cc:237] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:19.740978 8697 catalog_manager.cc:5484] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:19.896987 9732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:19.897260 9732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:19.900265 9732 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 19:56:19.997092 8970 raft_consensus.cc:1261] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:19.998045 9101 raft_consensus.cc:1261] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:19.998054 9100 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:19.998803 9704 consensus_queue.cc:1035] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:19.998943 9727 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:19.999033 9721 consensus_queue.cc:1035] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:20.005892 9764 mvcc.cc:205] Tried to move back new op lower bound from 7022974484461752320 to 7022974483408232448. Current Snapshot: MvccSnapshot[applied={T|T < 7022974484461752320}]
I20240501 19:56:20.006654 9763 mvcc.cc:205] Tried to move back new op lower bound from 7022974484461752320 to 7022974483408232448. Current Snapshot: MvccSnapshot[applied={T|T < 7022974484461752320}]
I20240501 19:56:20.021096 9770 mvcc.cc:205] Tried to move back new op lower bound from 7022974484460396544 to 7022974482662076416. Current Snapshot: MvccSnapshot[applied={T|T < 7022974484460396544}]
INSERT report
rows total: 2000
time total: 96.1015 ms
time per row: 0.0480507 ms
I20240501 19:56:20.045995 8697 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40084:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:20.052062 8819 tablet_service.cc:1467] Processing CreateTablet for tablet df35daed2f72400086ccd0b8c47ecab1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3c90b69a201147fa8697594a2940e3a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.052332 9080 tablet_service.cc:1467] Processing CreateTablet for tablet c869ad944b7a45d39ec171b0add850fb (DEFAULT_TABLE table=rebalance_test_table_1 [id=3c90b69a201147fa8697594a2940e3a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.052400 8818 tablet_service.cc:1467] Processing CreateTablet for tablet c869ad944b7a45d39ec171b0add850fb (DEFAULT_TABLE table=rebalance_test_table_1 [id=3c90b69a201147fa8697594a2940e3a2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.052445 9081 tablet_service.cc:1467] Processing CreateTablet for tablet df35daed2f72400086ccd0b8c47ecab1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3c90b69a201147fa8697594a2940e3a2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.052623 9080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c869ad944b7a45d39ec171b0add850fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.052846 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df35daed2f72400086ccd0b8c47ecab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.052399 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df35daed2f72400086ccd0b8c47ecab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.052932 8948 tablet_service.cc:1467] Processing CreateTablet for tablet 3211e49e29174e0dbf667959a78c30e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3c90b69a201147fa8697594a2940e3a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.053210 8948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3211e49e29174e0dbf667959a78c30e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.054795 9785 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:20.055078 8818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c869ad944b7a45d39ec171b0add850fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.055723 9787 tablet_bootstrap.cc:492] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:20.056221 9785 tablet_bootstrap.cc:654] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.056890 9787 tablet_bootstrap.cc:654] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.057659 9786 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:20.057804 9785 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:20.057773 8818 tablet_service.cc:1467] Processing CreateTablet for tablet 3211e49e29174e0dbf667959a78c30e2 (DEFAULT_TABLE table=rebalance_test_table_1 [id=3c90b69a201147fa8697594a2940e3a2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.057940 9785 ts_tablet_manager.cc:1367] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:20.058018 8818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3211e49e29174e0dbf667959a78c30e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.058405 9785 raft_consensus.cc:348] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.058455 9787 tablet_bootstrap.cc:492] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:20.058501 9785 raft_consensus.cc:374] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.058529 9785 raft_consensus.cc:729] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.058537 9787 ts_tablet_manager.cc:1367] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:20.058699 9785 consensus_queue.cc:260] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.058998 9786 tablet_bootstrap.cc:654] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.058956 9787 raft_consensus.cc:348] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.059052 9787 raft_consensus.cc:374] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.059077 9785 ts_tablet_manager.cc:1398] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:20.059084 9787 raft_consensus.cc:729] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.059235 9787 consensus_queue.cc:260] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.059288 9785 tablet_bootstrap.cc:492] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:20.059548 9787 ts_tablet_manager.cc:1398] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:20.061101 9785 tablet_bootstrap.cc:654] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.061633 9786 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:20.061726 9786 ts_tablet_manager.cc:1367] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:20.062192 9786 raft_consensus.cc:348] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.062281 9786 raft_consensus.cc:374] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.062317 9786 raft_consensus.cc:729] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.062395 9785 tablet_bootstrap.cc:492] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:20.062481 9785 ts_tablet_manager.cc:1367] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:20.062464 9786 consensus_queue.cc:260] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.062906 9785 raft_consensus.cc:348] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.062974 9785 raft_consensus.cc:374] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.062995 9785 raft_consensus.cc:729] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.063102 9785 consensus_queue.cc:260] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.063200 9786 ts_tablet_manager.cc:1398] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:20.063367 9786 tablet_bootstrap.cc:492] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:20.063377 9785 ts_tablet_manager.cc:1398] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:20.064257 9786 tablet_bootstrap.cc:654] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.065297 9786 tablet_bootstrap.cc:492] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:20.065367 9786 ts_tablet_manager.cc:1367] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:20.065742 9786 raft_consensus.cc:348] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.065821 9786 raft_consensus.cc:374] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.065842 9786 raft_consensus.cc:729] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.065969 9786 consensus_queue.cc:260] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.066270 9786 ts_tablet_manager.cc:1398] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:20.066661 9786 tablet_bootstrap.cc:492] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:20.067934 9786 tablet_bootstrap.cc:654] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.069095 9786 tablet_bootstrap.cc:492] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:20.069168 9786 ts_tablet_manager.cc:1367] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:20.069494 9786 raft_consensus.cc:348] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.069571 9786 raft_consensus.cc:374] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.069591 9786 raft_consensus.cc:729] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.069708 9786 consensus_queue.cc:260] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.069936 9786 ts_tablet_manager.cc:1398] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:20.112219 9704 raft_consensus.cc:482] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:20.112349 9704 raft_consensus.cc:504] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.112663 9704 leader_election.cc:288] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:20.113131 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3211e49e29174e0dbf667959a78c30e2" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b" is_pre_election: true
I20240501 19:56:20.113284 8970 raft_consensus.cc:2454] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 0.
I20240501 19:56:20.113612 8773 leader_election.cc:302] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:20.113790 9704 raft_consensus.cc:2781] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:20.113842 9704 raft_consensus.cc:482] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:20.113867 9704 raft_consensus.cc:3037] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.114696 9704 raft_consensus.cc:504] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.114951 9704 leader_election.cc:288] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Requested vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:20.115350 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3211e49e29174e0dbf667959a78c30e2" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
I20240501 19:56:20.115458 8970 raft_consensus.cc:3037] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.116330 8970 raft_consensus.cc:2454] T 3211e49e29174e0dbf667959a78c30e2 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 1.
I20240501 19:56:20.116616 8773 leader_election.cc:302] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:20.116808 9704 raft_consensus.cc:2781] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:20.116887 9704 raft_consensus.cc:686] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Running, Role: LEADER
I20240501 19:56:20.117060 9704 consensus_queue.cc:237] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.118695 8694 catalog_manager.cc:5484] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:20.303638 9704 raft_consensus.cc:482] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:20.303797 9704 raft_consensus.cc:504] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.304232 9704 leader_election.cc:288] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:20.308167 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df35daed2f72400086ccd0b8c47ecab1" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" is_pre_election: true
I20240501 19:56:20.308341 9101 raft_consensus.cc:2454] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 0.
I20240501 19:56:20.308665 8775 leader_election.cc:302] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:20.308852 9704 raft_consensus.cc:2781] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:20.308912 9704 raft_consensus.cc:482] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:20.308971 9704 raft_consensus.cc:3037] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.309804 9704 raft_consensus.cc:504] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.310066 9704 leader_election.cc:288] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Requested vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:20.310431 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df35daed2f72400086ccd0b8c47ecab1" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
I20240501 19:56:20.310555 9101 raft_consensus.cc:3037] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.311308 9101 raft_consensus.cc:2454] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 1.
I20240501 19:56:20.311630 8775 leader_election.cc:302] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:20.311810 9704 raft_consensus.cc:2781] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:20.311887 9704 raft_consensus.cc:686] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Running, Role: LEADER
I20240501 19:56:20.311993 9704 consensus_queue.cc:237] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.313540 8694 catalog_manager.cc:5484] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:20.537472 9768 raft_consensus.cc:482] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:20.537636 9768 raft_consensus.cc:504] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.538524 9768 leader_election.cc:288] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:20.541852 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c869ad944b7a45d39ec171b0add850fb" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982" is_pre_election: true
I20240501 19:56:20.542001 8839 raft_consensus.cc:2454] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 0.
I20240501 19:56:20.542371 9034 leader_election.cc:302] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:20.542598 9768 raft_consensus.cc:2781] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:20.542675 9768 raft_consensus.cc:482] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:20.542745 9768 raft_consensus.cc:3037] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.543499 9768 raft_consensus.cc:504] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.543795 9768 leader_election.cc:288] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Requested vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:20.544129 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c869ad944b7a45d39ec171b0add850fb" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
I20240501 19:56:20.544272 8839 raft_consensus.cc:3037] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.545220 8839 raft_consensus.cc:2454] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 1.
I20240501 19:56:20.545492 9034 leader_election.cc:302] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:20.545667 9768 raft_consensus.cc:2781] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:20.545822 9768 raft_consensus.cc:686] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Becoming Leader. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:20.546089 9768 consensus_queue.cc:237] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.548079 8694 catalog_manager.cc:5484] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67). New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:20.567852 9704 consensus_queue.cc:1035] T 3211e49e29174e0dbf667959a78c30e2 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 19:56:20.724426 9805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:20.724642 9805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:20.727552 9805 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 19:56:20.799575 9101 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:20.800328 9730 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:20.802963 8839 raft_consensus.cc:1261] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:20.803790 9768 consensus_queue.cc:1035] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 76.5814 ms
time per row: 0.0382907 ms
I20240501 19:56:20.844543 8694 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40084:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:20.850092 8948 tablet_service.cc:1467] Processing CreateTablet for tablet 49f4147edb7349bcbf51994e59501d19 (DEFAULT_TABLE table=rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.850327 9080 tablet_service.cc:1467] Processing CreateTablet for tablet 49f4147edb7349bcbf51994e59501d19 (DEFAULT_TABLE table=rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.850445 8948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f4147edb7349bcbf51994e59501d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.850430 9081 tablet_service.cc:1467] Processing CreateTablet for tablet aa477b2bd34d4b74b245314148e4851d (DEFAULT_TABLE table=rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.850446 8818 tablet_service.cc:1467] Processing CreateTablet for tablet ac9114fe81be4ba0bb4aadf28c61490d (DEFAULT_TABLE table=rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.850594 9080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f4147edb7349bcbf51994e59501d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.850690 8818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9114fe81be4ba0bb4aadf28c61490d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.850980 8949 tablet_service.cc:1467] Processing CreateTablet for tablet ac9114fe81be4ba0bb4aadf28c61490d (DEFAULT_TABLE table=rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.851176 8949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9114fe81be4ba0bb4aadf28c61490d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.852905 8950 tablet_service.cc:1467] Processing CreateTablet for tablet aa477b2bd34d4b74b245314148e4851d (DEFAULT_TABLE table=rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:20.853087 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa477b2bd34d4b74b245314148e4851d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.853168 8950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa477b2bd34d4b74b245314148e4851d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:20.853605 9849 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:20.853833 9850 tablet_bootstrap.cc:492] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:20.854770 9850 tablet_bootstrap.cc:654] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.854916 9848 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:20.855568 9848 tablet_bootstrap.cc:654] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.855579 9849 tablet_bootstrap.cc:654] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.856642 9848 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:20.856725 9848 ts_tablet_manager.cc:1367] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:20.856802 9850 tablet_bootstrap.cc:492] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:20.856884 9850 ts_tablet_manager.cc:1367] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:20.857177 9848 raft_consensus.cc:348] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.857286 9848 raft_consensus.cc:374] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.857319 9848 raft_consensus.cc:729] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.857456 9848 consensus_queue.cc:260] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.857484 9850 raft_consensus.cc:348] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.857560 9850 raft_consensus.cc:374] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.857604 9850 raft_consensus.cc:729] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.857734 9848 ts_tablet_manager.cc:1398] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:20.857728 9850 consensus_queue.cc:260] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.857906 9848 tablet_bootstrap.cc:492] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:20.857950 9850 ts_tablet_manager.cc:1398] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:20.858667 9848 tablet_bootstrap.cc:654] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.860249 9848 tablet_bootstrap.cc:492] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:20.860344 9848 ts_tablet_manager.cc:1367] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:20.860682 9849 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:20.860766 9849 ts_tablet_manager.cc:1367] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:20.860805 9848 raft_consensus.cc:348] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.860886 9848 raft_consensus.cc:374] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.860913 9848 raft_consensus.cc:729] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.861156 9848 consensus_queue.cc:260] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.861215 9849 raft_consensus.cc:348] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.861323 9849 raft_consensus.cc:374] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.861366 9849 raft_consensus.cc:729] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.861415 9848 ts_tablet_manager.cc:1398] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:20.861497 9849 consensus_queue.cc:260] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.861577 9848 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:20.861840 9849 ts_tablet_manager.cc:1398] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:20.861985 9849 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:20.862448 9848 tablet_bootstrap.cc:654] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.863639 9849 tablet_bootstrap.cc:654] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:20.864391 9848 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:20.864481 9848 ts_tablet_manager.cc:1367] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:20.864758 9849 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:20.864825 9849 ts_tablet_manager.cc:1367] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:20.864841 9848 raft_consensus.cc:348] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.864948 9848 raft_consensus.cc:374] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.864991 9848 raft_consensus.cc:729] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.865155 9848 consensus_queue.cc:260] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.865226 9849 raft_consensus.cc:348] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.865314 9849 raft_consensus.cc:374] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:20.865343 9849 raft_consensus.cc:729] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:20.865437 9848 ts_tablet_manager.cc:1398] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:20.865446 9849 consensus_queue.cc:260] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.865727 9849 ts_tablet_manager.cc:1398] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:20.870266 9800 raft_consensus.cc:482] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:20.870342 9800 raft_consensus.cc:504] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.870699 9800 leader_election.cc:288] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:20.874279 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f4147edb7349bcbf51994e59501d19" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b" is_pre_election: true
I20240501 19:56:20.874392 8970 raft_consensus.cc:2454] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 0.
I20240501 19:56:20.874660 9035 leader_election.cc:302] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:20.874838 9800 raft_consensus.cc:2781] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:20.874897 9800 raft_consensus.cc:482] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:20.874943 9800 raft_consensus.cc:3037] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.875725 9800 raft_consensus.cc:504] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.875977 9800 leader_election.cc:288] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Requested vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:20.876283 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f4147edb7349bcbf51994e59501d19" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
I20240501 19:56:20.876380 8970 raft_consensus.cc:3037] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.877010 8970 raft_consensus.cc:2454] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 1.
I20240501 19:56:20.877390 9035 leader_election.cc:302] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:20.877687 9800 raft_consensus.cc:2781] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:20.877876 9800 raft_consensus.cc:686] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Becoming Leader. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:20.878033 9800 consensus_queue.cc:237] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:20.879549 8693 catalog_manager.cc:5484] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67). New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:20.962404 9800 raft_consensus.cc:482] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:20.962532 9800 raft_consensus.cc:504] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.962890 9800 leader_election.cc:288] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:20.963130 9778 raft_consensus.cc:482] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:20.963228 9778 raft_consensus.cc:504] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.963366 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa477b2bd34d4b74b245314148e4851d" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b" is_pre_election: true
I20240501 19:56:20.963513 8970 raft_consensus.cc:2454] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 0.
I20240501 19:56:20.963668 9778 leader_election.cc:288] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:20.963829 9035 leader_election.cc:302] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:20.963982 9800 raft_consensus.cc:2781] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:20.963989 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa477b2bd34d4b74b245314148e4851d" candidate_uuid: "ac13bdb319c448dba072fb9e62d6893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" is_pre_election: true
I20240501 19:56:20.964087 9800 raft_consensus.cc:482] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:20.964180 9800 raft_consensus.cc:3037] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.964962 9800 raft_consensus.cc:504] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.965224 9101 raft_consensus.cc:2379] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ac13bdb319c448dba072fb9e62d6893b in current term 1: Already voted for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in this term.
I20240501 19:56:20.965251 9800 leader_election.cc:288] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Requested vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:20.965652 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa477b2bd34d4b74b245314148e4851d" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
I20240501 19:56:20.965744 8970 raft_consensus.cc:3037] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:20.966015 8906 leader_election.cc:302] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b; no voters: b3b45c9058d84b4e9ace42c9154ab0e0
I20240501 19:56:20.966477 8970 raft_consensus.cc:2454] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 1.
I20240501 19:56:20.966595 9778 raft_consensus.cc:2726] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 19:56:20.966753 9035 leader_election.cc:302] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:20.966958 9800 raft_consensus.cc:2781] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:20.967072 9800 raft_consensus.cc:686] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Becoming Leader. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:20.967203 9800 consensus_queue.cc:237] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:20.968677 8693 catalog_manager.cc:5484] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67). New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:21.289340 9778 raft_consensus.cc:482] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.289453 9778 raft_consensus.cc:504] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:21.289827 9778 leader_election.cc:288] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:21.293190 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9114fe81be4ba0bb4aadf28c61490d" candidate_uuid: "ac13bdb319c448dba072fb9e62d6893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982" is_pre_election: true
I20240501 19:56:21.293308 8839 raft_consensus.cc:2454] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac13bdb319c448dba072fb9e62d6893b in term 0.
I20240501 19:56:21.293743 8903 leader_election.cc:302] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:21.293934 9778 raft_consensus.cc:2781] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:21.294001 9778 raft_consensus.cc:482] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:21.294040 9778 raft_consensus.cc:3037] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.294870 9778 raft_consensus.cc:504] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:21.295114 9778 leader_election.cc:288] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 election: Requested vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:21.295408 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9114fe81be4ba0bb4aadf28c61490d" candidate_uuid: "ac13bdb319c448dba072fb9e62d6893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
I20240501 19:56:21.295517 8839 raft_consensus.cc:3037] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.296221 8839 raft_consensus.cc:2454] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac13bdb319c448dba072fb9e62d6893b in term 1.
I20240501 19:56:21.296468 8903 leader_election.cc:302] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:21.296608 9778 raft_consensus.cc:2781] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:21.296768 9778 raft_consensus.cc:686] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Becoming Leader. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Running, Role: LEADER
I20240501 19:56:21.296906 9778 consensus_queue.cc:237] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:21.298408 8693 catalog_manager.cc:5484] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac13bdb319c448dba072fb9e62d6893b (127.5.139.66). New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:21.411865 9863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:21.412086 9863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:21.415071 9863 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 19:56:21.485267 8839 raft_consensus.cc:1261] T ac9114fe81be4ba0bb4aadf28c61490d P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:21.485944 8970 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:21.486114 9778 consensus_queue.cc:1035] T ac9114fe81be4ba0bb4aadf28c61490d P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:21.486304 9800 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:21.495276 8970 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:21.496099 9800 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 85.0529 ms
time per row: 0.0425265 ms
I20240501 19:56:21.538411 8691 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40084:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:21.543428 8818 tablet_service.cc:1467] Processing CreateTablet for tablet 74706b357b2b43549757d24fa9a9d282 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a5200c5824e48fd9493fa155a0a6e47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.543781 8818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74706b357b2b43549757d24fa9a9d282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.543922 8817 tablet_service.cc:1467] Processing CreateTablet for tablet c63e353d2a544ce0af8551d8024743b7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a5200c5824e48fd9493fa155a0a6e47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.544169 8817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63e353d2a544ce0af8551d8024743b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.544211 9081 tablet_service.cc:1467] Processing CreateTablet for tablet 74706b357b2b43549757d24fa9a9d282 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a5200c5824e48fd9493fa155a0a6e47]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.544487 9081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74706b357b2b43549757d24fa9a9d282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.545279 8950 tablet_service.cc:1467] Processing CreateTablet for tablet c63e353d2a544ce0af8551d8024743b7 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a5200c5824e48fd9493fa155a0a6e47]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.545544 8950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63e353d2a544ce0af8551d8024743b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.546082 9910 tablet_bootstrap.cc:492] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:21.546806 9910 tablet_bootstrap.cc:654] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.547309 9911 tablet_bootstrap.cc:492] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:21.547824 9913 tablet_bootstrap.cc:492] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:21.547963 9080 tablet_service.cc:1467] Processing CreateTablet for tablet 58979b7e47734549bc48710ff1d2b1c1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a5200c5824e48fd9493fa155a0a6e47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.548116 9911 tablet_bootstrap.cc:654] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.548202 9080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58979b7e47734549bc48710ff1d2b1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.548461 9913 tablet_bootstrap.cc:654] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.549172 9911 tablet_bootstrap.cc:492] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:21.549167 9910 tablet_bootstrap.cc:492] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:21.549237 9911 ts_tablet_manager.cc:1367] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:21.549248 9910 ts_tablet_manager.cc:1367] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:21.549674 9911 raft_consensus.cc:348] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.549677 9910 raft_consensus.cc:348] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } }
I20240501 19:56:21.549772 9911 raft_consensus.cc:374] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.549782 9910 raft_consensus.cc:374] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.549818 9910 raft_consensus.cc:729] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.549810 9911 raft_consensus.cc:729] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.549940 9910 consensus_queue.cc:260] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } }
I20240501 19:56:21.549933 9911 consensus_queue.cc:260] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.550207 9911 ts_tablet_manager.cc:1398] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:21.550210 9910 ts_tablet_manager.cc:1398] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:21.550352 9911 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:21.550367 9910 tablet_bootstrap.cc:492] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:21.551240 9910 tablet_bootstrap.cc:654] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.543727 8819 tablet_service.cc:1467] Processing CreateTablet for tablet 58979b7e47734549bc48710ff1d2b1c1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8a5200c5824e48fd9493fa155a0a6e47]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:21.551908 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58979b7e47734549bc48710ff1d2b1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:21.553288 9911 tablet_bootstrap.cc:654] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.555549 9910 tablet_bootstrap.cc:492] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:21.555672 9910 ts_tablet_manager.cc:1367] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:21.556030 9910 raft_consensus.cc:348] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.556119 9910 raft_consensus.cc:374] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.556154 9910 raft_consensus.cc:729] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.556281 9910 consensus_queue.cc:260] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.556555 9910 ts_tablet_manager.cc:1398] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:21.556751 9910 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:21.557637 9910 tablet_bootstrap.cc:654] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:21.558081 9911 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:21.558169 9911 ts_tablet_manager.cc:1367] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:21.558610 9911 raft_consensus.cc:348] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.558715 9911 raft_consensus.cc:374] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.558754 9911 raft_consensus.cc:729] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.558926 9911 consensus_queue.cc:260] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.559264 9911 ts_tablet_manager.cc:1398] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:21.559402 9910 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:21.559532 9910 ts_tablet_manager.cc:1367] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:21.560060 9913 tablet_bootstrap.cc:492] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:21.560046 9910 raft_consensus.cc:348] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.560170 9913 ts_tablet_manager.cc:1367] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.000s
I20240501 19:56:21.560262 9910 raft_consensus.cc:374] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.560318 9910 raft_consensus.cc:729] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.560441 9910 consensus_queue.cc:260] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.560712 9913 raft_consensus.cc:348] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } }
I20240501 19:56:21.560794 9910 ts_tablet_manager.cc:1398] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:21.560819 9913 raft_consensus.cc:374] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:21.560850 9913 raft_consensus.cc:729] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:21.560972 9913 consensus_queue.cc:260] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } }
I20240501 19:56:21.561264 9913 ts_tablet_manager.cc:1398] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:21.571864 9730 raft_consensus.cc:482] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.571952 9730 raft_consensus.cc:504] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:21.572216 9730 leader_election.cc:288] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14798909ff724444a8be03c45c41929a (127.5.139.71:38393)
W20240501 19:56:21.573426 8773 leader_election.cc:334] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 14798909ff724444a8be03c45c41929a (127.5.139.71:38393): Network error: Client connection negotiation failed: client connection to 127.5.139.71:38393: connect: Connection refused (error 111)
I20240501 19:56:21.573484 8773 leader_election.cc:302] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982; no voters: 14798909ff724444a8be03c45c41929a
I20240501 19:56:21.573626 9730 raft_consensus.cc:2726] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 19:56:21.750736 9901 raft_consensus.cc:482] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.750873 9901 raft_consensus.cc:504] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.751194 9901 leader_election.cc:288] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:21.751668 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58979b7e47734549bc48710ff1d2b1c1" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982" is_pre_election: true
I20240501 19:56:21.751813 8839 raft_consensus.cc:2454] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 0.
I20240501 19:56:21.752144 9034 leader_election.cc:302] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:21.752312 9901 raft_consensus.cc:2781] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:21.752370 9901 raft_consensus.cc:482] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:21.752418 9901 raft_consensus.cc:3037] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.753329 9901 raft_consensus.cc:504] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.753579 9901 leader_election.cc:288] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Requested vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:21.753993 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58979b7e47734549bc48710ff1d2b1c1" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
I20240501 19:56:21.754105 8839 raft_consensus.cc:3037] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.754842 8839 raft_consensus.cc:2454] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 1.
I20240501 19:56:21.755122 9034 leader_election.cc:302] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:21.755270 9901 raft_consensus.cc:2781] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:21.755363 9901 raft_consensus.cc:686] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Becoming Leader. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:21.755498 9901 consensus_queue.cc:237] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.757174 8696 catalog_manager.cc:5484] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67). New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:21.777120 9901 raft_consensus.cc:482] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.777266 9901 raft_consensus.cc:504] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.777627 9901 leader_election.cc:288] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:21.778074 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74706b357b2b43549757d24fa9a9d282" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982" is_pre_election: true
I20240501 19:56:21.778204 8839 raft_consensus.cc:2454] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 0.
I20240501 19:56:21.778441 9034 leader_election.cc:302] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:21.778592 9901 raft_consensus.cc:2781] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:21.778649 9901 raft_consensus.cc:482] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:21.778688 9901 raft_consensus.cc:3037] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.779564 9901 raft_consensus.cc:504] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.779824 9901 leader_election.cc:288] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Requested vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:21.780122 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "74706b357b2b43549757d24fa9a9d282" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
I20240501 19:56:21.780226 8839 raft_consensus.cc:3037] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.780958 8839 raft_consensus.cc:2454] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 1.
I20240501 19:56:21.781244 9034 leader_election.cc:302] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:21.781420 9901 raft_consensus.cc:2781] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:21.781515 9901 raft_consensus.cc:686] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Becoming Leader. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:21.781636 9901 consensus_queue.cc:237] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:21.783139 8696 catalog_manager.cc:5484] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67). New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:21.802670 9730 raft_consensus.cc:482] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:21.802774 9730 raft_consensus.cc:504] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } }
I20240501 19:56:21.803046 9730 leader_election.cc:288] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:21.803474 8969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63e353d2a544ce0af8551d8024743b7" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b" is_pre_election: true
I20240501 19:56:21.803606 8969 raft_consensus.cc:2454] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 0.
I20240501 19:56:21.803864 8773 leader_election.cc:302] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:21.804054 9730 raft_consensus.cc:2781] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:21.804107 9730 raft_consensus.cc:482] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:21.804134 9730 raft_consensus.cc:3037] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.804900 9730 raft_consensus.cc:504] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } }
I20240501 19:56:21.805151 9730 leader_election.cc:288] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Requested vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:21.805476 8969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63e353d2a544ce0af8551d8024743b7" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
I20240501 19:56:21.805585 8969 raft_consensus.cc:3037] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:21.806275 8969 raft_consensus.cc:2454] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 1.
I20240501 19:56:21.806581 8773 leader_election.cc:302] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:21.806752 9730 raft_consensus.cc:2781] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:21.806902 9730 raft_consensus.cc:686] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Running, Role: LEADER
I20240501 19:56:21.807040 9730 consensus_queue.cc:237] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } }
I20240501 19:56:21.808554 8696 catalog_manager.cc:5484] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:21.935333 9925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:21.935600 9925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:21.938638 9925 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 19:56:22.020251 8839 raft_consensus.cc:1261] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.020665 9901 consensus_queue.cc:1035] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.022948 8969 raft_consensus.cc:1261] T c63e353d2a544ce0af8551d8024743b7 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.024803 9923 consensus_queue.cc:1035] T c63e353d2a544ce0af8551d8024743b7 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.025859 8838 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.026515 9902 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 88.6643 ms
time per row: 0.0443322 ms
I20240501 19:56:22.068029 8691 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40084:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:22.072968 8950 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca216ce0c6a420f8c90273a02ad8cf4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.073019 9080 tablet_service.cc:1467] Processing CreateTablet for tablet 0ca216ce0c6a420f8c90273a02ad8cf4 (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.073311 8950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca216ce0c6a420f8c90273a02ad8cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.073312 9080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca216ce0c6a420f8c90273a02ad8cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.073608 8948 tablet_service.cc:1467] Processing CreateTablet for tablet d827d0b0d5ba4969908768c96f92c1f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.073853 8948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d827d0b0d5ba4969908768c96f92c1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.075541 8949 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1ab4dc3bb841c2b352cee4ec7aa373 (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.075700 9977 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:22.075742 8949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1ab4dc3bb841c2b352cee4ec7aa373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.076439 9977 tablet_bootstrap.cc:654] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.077443 9080 tablet_service.cc:1467] Processing CreateTablet for tablet 0f1ab4dc3bb841c2b352cee4ec7aa373 (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.077690 9080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1ab4dc3bb841c2b352cee4ec7aa373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.077755 8819 tablet_service.cc:1467] Processing CreateTablet for tablet d827d0b0d5ba4969908768c96f92c1f3 (DEFAULT_TABLE table=rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:22.077836 9976 tablet_bootstrap.cc:492] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:22.078024 8819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d827d0b0d5ba4969908768c96f92c1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:22.078606 9976 tablet_bootstrap.cc:654] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.077469 9977 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:22.079166 9977 ts_tablet_manager.cc:1367] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:22.079587 9977 raft_consensus.cc:348] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:22.079692 9977 raft_consensus.cc:374] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.079735 9977 raft_consensus.cc:729] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.079741 9976 tablet_bootstrap.cc:492] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:22.079818 9976 ts_tablet_manager.cc:1367] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:22.079890 9977 consensus_queue.cc:260] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:22.080258 9976 raft_consensus.cc:348] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.080333 9980 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: Bootstrap starting.
I20240501 19:56:22.080355 9976 raft_consensus.cc:374] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.080390 9976 raft_consensus.cc:729] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.080540 9976 consensus_queue.cc:260] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.080596 9977 ts_tablet_manager.cc:1398] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.080757 9977 tablet_bootstrap.cc:492] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:22.080814 9976 ts_tablet_manager.cc:1398] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.080976 9976 tablet_bootstrap.cc:492] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0: Bootstrap starting.
I20240501 19:56:22.081137 9980 tablet_bootstrap.cc:654] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.081627 9977 tablet_bootstrap.cc:654] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.081835 9976 tablet_bootstrap.cc:654] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.082101 9980 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: No bootstrap required, opened a new log
I20240501 19:56:22.082175 9980 ts_tablet_manager.cc:1367] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:22.082600 9980 raft_consensus.cc:348] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:22.082710 9980 raft_consensus.cc:374] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.082749 9980 raft_consensus.cc:729] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.082862 9980 consensus_queue.cc:260] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:22.082978 9977 tablet_bootstrap.cc:492] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:22.083050 9977 ts_tablet_manager.cc:1367] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:22.083110 9980 ts_tablet_manager.cc:1398] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:22.083503 9977 raft_consensus.cc:348] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.083629 9977 raft_consensus.cc:374] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.083665 9977 raft_consensus.cc:729] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.083784 9977 consensus_queue.cc:260] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.083861 9976 tablet_bootstrap.cc:492] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0: No bootstrap required, opened a new log
I20240501 19:56:22.083925 9976 ts_tablet_manager.cc:1367] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:22.084031 9977 ts_tablet_manager.cc:1398] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.084160 9977 tablet_bootstrap.cc:492] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b: Bootstrap starting.
I20240501 19:56:22.084378 9976 raft_consensus.cc:348] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.084470 9976 raft_consensus.cc:374] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.084507 9976 raft_consensus.cc:729] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.084621 9976 consensus_queue.cc:260] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.084888 9976 ts_tablet_manager.cc:1398] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:22.085126 9977 tablet_bootstrap.cc:654] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:22.086550 9977 tablet_bootstrap.cc:492] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b: No bootstrap required, opened a new log
I20240501 19:56:22.089094 9977 ts_tablet_manager.cc:1367] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 19:56:22.089444 9977 raft_consensus.cc:348] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.089522 9977 raft_consensus.cc:374] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:22.089555 9977 raft_consensus.cc:729] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Initialized, Role: FOLLOWER
I20240501 19:56:22.089700 9977 consensus_queue.cc:260] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.090021 9977 ts_tablet_manager.cc:1398] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:22.344292 9923 raft_consensus.cc:482] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:22.344422 9923 raft_consensus.cc:504] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:22.344779 9923 leader_election.cc:288] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:22.345329 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b" is_pre_election: true
I20240501 19:56:22.345471 8970 raft_consensus.cc:2454] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 0.
I20240501 19:56:22.345786 8773 leader_election.cc:302] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:22.345960 9923 raft_consensus.cc:2781] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:22.346016 9923 raft_consensus.cc:482] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:22.346045 9923 raft_consensus.cc:3037] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.346825 9923 raft_consensus.cc:504] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:22.347069 9923 leader_election.cc:288] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Requested vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:22.347393 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3" candidate_uuid: "0f2b49c6d3dc49fabee40802e73ac982" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
I20240501 19:56:22.347496 8970 raft_consensus.cc:3037] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.348229 8970 raft_consensus.cc:2454] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f2b49c6d3dc49fabee40802e73ac982 in term 1.
I20240501 19:56:22.348476 8773 leader_election.cc:302] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:22.348631 9923 raft_consensus.cc:2781] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:22.348708 9923 raft_consensus.cc:686] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Becoming Leader. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Running, Role: LEADER
I20240501 19:56:22.348842 9923 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } }
I20240501 19:56:22.350355 8696 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:22.364421 9778 raft_consensus.cc:482] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:22.364542 9778 raft_consensus.cc:504] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.364881 9778 leader_election.cc:288] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:22.365366 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca216ce0c6a420f8c90273a02ad8cf4" candidate_uuid: "ac13bdb319c448dba072fb9e62d6893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" is_pre_election: true
I20240501 19:56:22.365509 9101 raft_consensus.cc:2454] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ac13bdb319c448dba072fb9e62d6893b in term 0.
I20240501 19:56:22.365954 8906 leader_election.cc:302] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:22.366139 9778 raft_consensus.cc:2781] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:22.366214 9778 raft_consensus.cc:482] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:22.366266 9778 raft_consensus.cc:3037] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.367422 9778 raft_consensus.cc:504] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.367631 9778 leader_election.cc:288] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 election: Requested vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:22.367981 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ca216ce0c6a420f8c90273a02ad8cf4" candidate_uuid: "ac13bdb319c448dba072fb9e62d6893b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
I20240501 19:56:22.368141 9101 raft_consensus.cc:3037] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.368942 9101 raft_consensus.cc:2454] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ac13bdb319c448dba072fb9e62d6893b in term 1.
I20240501 19:56:22.369467 8906 leader_election.cc:302] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:22.369853 9778 raft_consensus.cc:2781] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:22.370029 9778 raft_consensus.cc:686] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Becoming Leader. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Running, Role: LEADER
I20240501 19:56:22.370184 9778 consensus_queue.cc:237] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.373602 8696 catalog_manager.cc:5484] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: term changed from 0 to 1, leader changed from <none> to ac13bdb319c448dba072fb9e62d6893b (127.5.139.66). New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:22.414652 9966 raft_consensus.cc:482] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:22.414793 9966 raft_consensus.cc:504] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.415169 9966 leader_election.cc:288] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:22.415649 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ab4dc3bb841c2b352cee4ec7aa373" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b" is_pre_election: true
I20240501 19:56:22.415773 8970 raft_consensus.cc:2454] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 0.
I20240501 19:56:22.416064 9035 leader_election.cc:302] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:22.416239 9966 raft_consensus.cc:2781] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:22.416297 9966 raft_consensus.cc:482] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:22.416334 9966 raft_consensus.cc:3037] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.417177 9966 raft_consensus.cc:504] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.417451 9966 leader_election.cc:288] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Requested vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:22.417784 8970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f1ab4dc3bb841c2b352cee4ec7aa373" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
I20240501 19:56:22.417897 8970 raft_consensus.cc:3037] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:22.418587 8970 raft_consensus.cc:2454] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 1.
I20240501 19:56:22.418871 9035 leader_election.cc:302] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: ac13bdb319c448dba072fb9e62d6893b, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:22.419009 9966 raft_consensus.cc:2781] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:22.419090 9966 raft_consensus.cc:686] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Becoming Leader. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:22.419236 9966 consensus_queue.cc:237] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } }
I20240501 19:56:22.420867 8696 catalog_manager.cc:5484] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67). New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:22.540565 9989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:22.540781 9989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:22.543848 9989 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 19:56:22.621703 9101 raft_consensus.cc:1261] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.622113 9778 consensus_queue.cc:1035] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.623685 8970 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.624169 9923 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:22.629115 8969 raft_consensus.cc:1261] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:22.629446 9966 consensus_queue.cc:1035] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 86.23 ms
time per row: 0.043115 ms
I20240501 19:56:22.669579 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.68:46413
--local_ip_for_outbound_sockets=127.5.139.68
--tserver_master_addrs=127.5.139.126:34427
--webserver_port=37379
--webserver_interface=127.5.139.68
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:22.771792 10039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:22.771976 10039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:22.772017 10039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:22.774952 10039 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:22.775007 10039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:22.775103 10039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.68
I20240501 19:56:22.778556 10039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.68:46413
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.139.68
--webserver_port=37379
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.10039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.68
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:22.779299 10039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:22.780301 10039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:22.788152 10045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:22.788568 10047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:22.789196 10044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:22.789789 10039 server_base.cc:1034] running on GCE node
I20240501 19:56:22.790107 10039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:22.790585 10039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:22.791728 10039 hybrid_clock.cc:648] HybridClock initialized: now 1714593382791717 us; error 24 us; skew 500 ppm
I20240501 19:56:22.793429 10039 webserver.cc:460] Webserver started at http://127.5.139.68:37379/ using document root <none> and password file <none>
I20240501 19:56:22.793962 10039 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:22.794039 10039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:22.797032 10039 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:22.798740 10053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:22.799763 10039 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:22.799867 10039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c9564f3b39b549c1bd995354fa135c5f"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:22.800236 10039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:22.821588 10039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:22.822183 10039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:22.822332 10039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:22.822850 10039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:22.823701 10039 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:22.823750 10039 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:22.823807 10039 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:22.823832 10039 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:22.832194 10039 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.68:46413
I20240501 19:56:22.832219 10166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.68:46413 every 8 connection(s)
I20240501 19:56:22.833163 10039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:22.834648 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 10039
I20240501 19:56:22.835062 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.69:42483
--local_ip_for_outbound_sockets=127.5.139.69
--tserver_master_addrs=127.5.139.126:34427
--webserver_port=42907
--webserver_interface=127.5.139.69
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:22.843088 10167 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:22.843300 10167 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:22.843873 10167 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:22.844938 8696 ts_manager.cc:194] Re-registered known tserver with Master: c9564f3b39b549c1bd995354fa135c5f (127.5.139.68:46413)
I20240501 19:56:22.845784 8696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.68:40159
W20240501 19:56:22.938786 10170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:22.938968 10170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:22.939007 10170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:22.941794 10170 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:22.941843 10170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:22.941903 10170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.69
I20240501 19:56:22.945452 10170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.69:42483
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.139.69
--webserver_port=42907
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.10170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.69
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:22.946171 10170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:22.947221 10170 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:22.954910 10176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:22.955493 10175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:22.956099 10178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:22.956789 10170 server_base.cc:1034] running on GCE node
I20240501 19:56:22.957255 10170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:22.957934 10170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:22.959113 10170 hybrid_clock.cc:648] HybridClock initialized: now 1714593382959092 us; error 38 us; skew 500 ppm
I20240501 19:56:22.960837 10170 webserver.cc:460] Webserver started at http://127.5.139.69:42907/ using document root <none> and password file <none>
I20240501 19:56:22.961357 10170 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:22.961428 10170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:22.964521 10170 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:22.966229 10184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:22.967068 10170 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:22.967161 10170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
format_stamp: "Formatted at 2024-05-01 19:56:14 on dist-test-slave-twc7"
I20240501 19:56:22.967518 10170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:22.986266 10170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:22.986814 10170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:22.986951 10170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:22.987437 10170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:22.988265 10170 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:22.988323 10170 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:22.988379 10170 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:22.988425 10170 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:22.996243 10170 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.69:42483
I20240501 19:56:22.996266 10297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.69:42483 every 8 connection(s)
I20240501 19:56:22.997196 10170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:23.000524 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 10170
I20240501 19:56:23.001008 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.70:43441
--local_ip_for_outbound_sockets=127.5.139.70
--tserver_master_addrs=127.5.139.126:34427
--webserver_port=40021
--webserver_interface=127.5.139.70
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:23.006930 10298 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:23.007170 10298 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:23.007815 10298 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:23.008818 8696 ts_manager.cc:194] Re-registered known tserver with Master: 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483)
I20240501 19:56:23.009373 8696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.69:37479
W20240501 19:56:23.109560 10301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:23.109805 10301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:23.109870 10301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:23.112757 10301 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:23.112823 10301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:23.112910 10301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.70
I20240501 19:56:23.116412 10301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.70:43441
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.139.70
--webserver_port=40021
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.10301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.70
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:23.117249 10301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:23.118307 10301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:23.126161 10306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:23.126739 10309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:23.127338 10307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:23.127921 10301 server_base.cc:1034] running on GCE node
I20240501 19:56:23.128259 10301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:23.128732 10301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:23.129865 10301 hybrid_clock.cc:648] HybridClock initialized: now 1714593383129850 us; error 34 us; skew 500 ppm
I20240501 19:56:23.131534 10301 webserver.cc:460] Webserver started at http://127.5.139.70:40021/ using document root <none> and password file <none>
I20240501 19:56:23.132051 10301 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:23.132123 10301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:23.136617 10301 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:23.138453 10315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:23.139395 10301 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:23.139492 10301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "6b4f42453a864314adc86aba2c3b3556"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-twc7"
I20240501 19:56:23.139897 10301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:23.158695 10301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:23.159340 10301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:23.159482 10301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:23.160028 10301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:23.160848 10301 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:23.160893 10301 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:23.160948 10301 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:23.160974 10301 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:23.169028 10301 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.70:43441
I20240501 19:56:23.169095 10428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.70:43441 every 8 connection(s)
I20240501 19:56:23.169993 10301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:23.177634 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 10301
I20240501 19:56:23.178081 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.71:38393
--local_ip_for_outbound_sockets=127.5.139.71
--tserver_master_addrs=127.5.139.126:34427
--webserver_port=34883
--webserver_interface=127.5.139.71
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:23.178449 10429 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:23.178649 10429 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:23.179104 10429 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:23.180022 8696 ts_manager.cc:194] Re-registered known tserver with Master: 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441)
I20240501 19:56:23.180519 8696 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.70:40521
W20240501 19:56:23.280592 10432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:23.280777 10432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:23.280849 10432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:23.283854 10432 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:23.283916 10432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:23.284001 10432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.71
I20240501 19:56:23.287497 10432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.71:38393
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.139.71
--webserver_port=34883
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.10432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.71
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:23.288261 10432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:23.289319 10432 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:23.297130 10437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:23.297549 10438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:23.298084 10440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:23.298766 10432 server_base.cc:1034] running on GCE node
I20240501 19:56:23.299098 10432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:23.299623 10432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:23.300788 10432 hybrid_clock.cc:648] HybridClock initialized: now 1714593383300753 us; error 35 us; skew 500 ppm
I20240501 19:56:23.302613 10432 webserver.cc:460] Webserver started at http://127.5.139.71:34883/ using document root <none> and password file <none>
I20240501 19:56:23.303139 10432 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:23.303211 10432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:23.306562 10432 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:23.308425 10446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:23.309374 10432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:23.309475 10432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "14798909ff724444a8be03c45c41929a"
format_stamp: "Formatted at 2024-05-01 19:56:15 on dist-test-slave-twc7"
I20240501 19:56:23.309864 10432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:23.331604 10432 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:23.332197 10432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:23.332337 10432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:23.332823 10432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:23.333889 10453 ts_tablet_manager.cc:516] Loading tablet metadata (0/1 complete)
I20240501 19:56:23.336761 10432 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 19:56:23.336813 10432 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20240501 19:56:23.336861 10432 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 19:56:23.338229 10432 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 19:56:23.338271 10432 ts_tablet_manager.cc:569] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:23.338471 10453 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Bootstrap starting.
I20240501 19:56:23.348045 10432 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.71:38393
I20240501 19:56:23.348083 10560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.71:38393 every 8 connection(s)
I20240501 19:56:23.348940 10432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:23.351661 10453 log.cc:824] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:23.353577 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 10432
I20240501 19:56:23.353700 10453 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:23.353749 5677 test_util.cc:268] Using random seed: -510882412
I20240501 19:56:23.354130 10453 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Bootstrap complete.
I20240501 19:56:23.355415 10453 ts_tablet_manager.cc:1367] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20240501 19:56:23.357616 10561 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:23.357822 10561 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:23.358306 10561 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:23.358507 10453 raft_consensus.cc:348] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:23.359057 10453 raft_consensus.cc:729] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14798909ff724444a8be03c45c41929a, State: Initialized, Role: FOLLOWER
I20240501 19:56:23.359464 10453 consensus_queue.cc:260] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:23.360164 10453 ts_tablet_manager.cc:1398] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20240501 19:56:23.361541 8693 ts_manager.cc:194] Re-registered known tserver with Master: 14798909ff724444a8be03c45c41929a (127.5.139.71:38393)
I20240501 19:56:23.362609 8693 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.71:45863
I20240501 19:56:23.364676 10561 heartbeater.cc:503] Master 127.5.139.126:34427 was elected leader, sending a full tablet report...
I20240501 19:56:23.365446 5677 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 19:56:23.365710 5677 test_util.cc:268] Using random seed: -510870455
I20240501 19:56:23.374300 5677 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 19:56:23.374478 5677 test_util.cc:268] Using random seed: -510861684
I20240501 19:56:23.383592 5677 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 19:56:23.383754 5677 test_util.cc:268] Using random seed: -510852409
I20240501 19:56:23.402640 5677 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 19:56:23.402781 5677 test_util.cc:268] Using random seed: -510833381
I20240501 19:56:23.412487 5677 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
W20240501 19:56:23.600500 10562 tablet.cc:2369] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:23.819887 10232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:23.828423 10495 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:23.828929 8949 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:23.839195 8819 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:23.840390 9080 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:23.847716 10167 heartbeater.cc:503] Master 127.5.139.126:34427 was elected leader, sending a full tablet report...
I20240501 19:56:23.855404 10101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:23.862452 10363 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:24.011554 10298 heartbeater.cc:503] Master 127.5.139.126:34427 was elected leader, sending a full tablet report...
I20240501 19:56:24.124917 8819 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:24.128563 10495 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:24.136129 10363 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:24.139938 10232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:24.145958 9080 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:24.170071 10101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:24.182299 10429 heartbeater.cc:503] Master 127.5.139.126:34427 was elected leader, sending a full tablet report...
I20240501 19:56:24.201750 8949 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:24.291802 9098 consensus_queue.cc:237] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 27, Committed index: 27, Last appended: 1.27, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:24.297156 8839 raft_consensus.cc:1261] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 27. Preceding OpId from leader: term: 1 index: 29. (index mismatch)
I20240501 19:56:24.297680 10646 consensus_queue.cc:1035] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 28, Last known committed idx: 27, Time since last communication: 0.000s
I20240501 19:56:24.301172 8839 raft_consensus.cc:2932] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
I20240501 19:56:24.300767 10734 raft_consensus.cc:2932] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.28: config changed from index -1 to 28, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New config: { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
I20240501 19:56:24.304212 8696 catalog_manager.cc:5484] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index -1 to 28, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
W20240501 19:56:24.307641 9034 consensus_peers.cc:485] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 -> Peer c9564f3b39b549c1bd995354fa135c5f (127.5.139.68:46413): Couldn't send request to peer c9564f3b39b549c1bd995354fa135c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 74706b357b2b43549757d24fa9a9d282. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.340999 9098 consensus_queue.cc:237] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 41, Committed index: 41, Last appended: 1.41, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:24.342818 8838 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 41. Preceding OpId from leader: term: 1 index: 42. (index mismatch)
I20240501 19:56:24.344507 10734 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 42, Last known committed idx: 41, Time since last communication: 0.000s
W20240501 19:56:24.348289 9034 consensus_peers.cc:485] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 -> Peer 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441): Couldn't send request to peer 6b4f42453a864314adc86aba2c3b3556. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58979b7e47734549bc48710ff1d2b1c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.348721 9902 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.42: config changed from index -1 to 42, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } } }
I20240501 19:56:24.349774 8838 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Committing config change with OpId 1.42: config changed from index -1 to 42, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New config: { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } } }
I20240501 19:56:24.351708 8696 catalog_manager.cc:5484] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index -1 to 42, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:24.395648 8838 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 43, Committed index: 43, Last appended: 1.44, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:24.397547 8969 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 44. Preceding OpId from leader: term: 1 index: 45. (index mismatch)
I20240501 19:56:24.398095 9923 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 45, Last known committed idx: 43, Time since last communication: 0.000s
I20240501 19:56:24.399993 10649 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:24.402707 8696 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index -1 to 45, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:24.409256 8967 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Committing config change with OpId 1.45: config changed from index -1 to 45, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
W20240501 19:56:24.410941 8772 consensus_peers.cc:485] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 -> Peer 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483): Couldn't send request to peer 6977f9a52e9b4d06bcb12f9bee61553b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d827d0b0d5ba4969908768c96f92c1f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.424216 8970 consensus_queue.cc:237] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 1.23, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } }
I20240501 19:56:24.426862 9098 raft_consensus.cc:1261] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 23. Preceding OpId from leader: term: 1 index: 24. (index mismatch)
I20240501 19:56:24.427186 10615 consensus_queue.cc:1035] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20240501 19:56:24.428737 10615 raft_consensus.cc:2932] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER 14798909ff724444a8be03c45c41929a (127.5.139.71) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } } }
I20240501 19:56:24.433338 9098 raft_consensus.cc:2932] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Committing config change with OpId 1.24: config changed from index -1 to 24, NON_VOTER 14798909ff724444a8be03c45c41929a (127.5.139.71) added. New config: { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } } }
I20240501 19:56:24.435700 8696 catalog_manager.cc:5484] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index -1 to 24, NON_VOTER 14798909ff724444a8be03c45c41929a (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } } }
W20240501 19:56:24.461485 8904 consensus_peers.cc:485] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b -> Peer 14798909ff724444a8be03c45c41929a (127.5.139.71:38393): Couldn't send request to peer 14798909ff724444a8be03c45c41929a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ca216ce0c6a420f8c90273a02ad8cf4. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.469225 9098 consensus_queue.cc:237] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:24.470788 8839 raft_consensus.cc:1261] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
W20240501 19:56:24.471098 9034 consensus_peers.cc:485] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 -> Peer 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441): Couldn't send request to peer 6b4f42453a864314adc86aba2c3b3556. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c869ad944b7a45d39ec171b0add850fb. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.471411 10646 consensus_queue.cc:1035] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
I20240501 19:56:24.472772 10646 raft_consensus.cc:2932] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } } }
I20240501 19:56:24.475381 8696 catalog_manager.cc:5484] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index -1 to 46, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:24.476238 8839 raft_consensus.cc:2932] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } } }
I20240501 19:56:24.488298 9098 consensus_queue.cc:237] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 1.25, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:24.489928 8969 raft_consensus.cc:1261] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 25. Preceding OpId from leader: term: 1 index: 26. (index mismatch)
W20240501 19:56:24.490309 9034 consensus_peers.cc:485] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 -> Peer c9564f3b39b549c1bd995354fa135c5f (127.5.139.68:46413): Couldn't send request to peer c9564f3b39b549c1bd995354fa135c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f1ab4dc3bb841c2b352cee4ec7aa373. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.490437 10646 consensus_queue.cc:1035] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
I20240501 19:56:24.493369 10734 raft_consensus.cc:2932] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
I20240501 19:56:24.495029 8969 raft_consensus.cc:2932] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Committing config change with OpId 1.26: config changed from index -1 to 26, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New config: { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
I20240501 19:56:24.497534 8694 catalog_manager.cc:5484] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index -1 to 26, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
I20240501 19:56:24.551607 8838 consensus_queue.cc:237] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
W20240501 19:56:24.553090 8772 consensus_peers.cc:485] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 -> Peer 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483): Couldn't send request to peer 6977f9a52e9b4d06bcb12f9bee61553b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c4e313b03cc44d0a2f7864f7e9d4f90. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.554235 8970 raft_consensus.cc:1261] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240501 19:56:24.554916 9923 consensus_queue.cc:1035] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20240501 19:56:24.561535 10738 raft_consensus.cc:2932] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index -1 to 49, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:24.564747 8693 catalog_manager.cc:5484] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index -1 to 49, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:24.570951 8969 raft_consensus.cc:2932] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index -1 to 49, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:24.594070 9101 consensus_queue.cc:237] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 58, Committed index: 58, Last appended: 1.58, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } }
W20240501 19:56:24.602614 9035 consensus_peers.cc:485] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 -> Peer 14798909ff724444a8be03c45c41929a (127.5.139.71:38393): Couldn't send request to peer 14798909ff724444a8be03c45c41929a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49f4147edb7349bcbf51994e59501d19. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.605260 8967 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 58. Preceding OpId from leader: term: 1 index: 59. (index mismatch)
I20240501 19:56:24.605630 10734 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 59, Last known committed idx: 58, Time since last communication: 0.000s
I20240501 19:56:24.609328 10734 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER 14798909ff724444a8be03c45c41929a (127.5.139.71) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } } }
I20240501 19:56:24.612088 8693 catalog_manager.cc:5484] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index -1 to 59, NON_VOTER 14798909ff724444a8be03c45c41929a (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:24.613750 8969 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Committing config change with OpId 1.59: config changed from index -1 to 59, NON_VOTER 14798909ff724444a8be03c45c41929a (127.5.139.71) added. New config: { opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } } }
I20240501 19:56:24.648653 8967 consensus_queue.cc:237] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 44, Committed index: 44, Last appended: 1.45, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
W20240501 19:56:24.655066 8903 consensus_peers.cc:485] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b -> Peer 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441): Couldn't send request to peer 6b4f42453a864314adc86aba2c3b3556. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68c8311312bd4b50bdbedf414ada6bd2. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.656380 9101 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 45. Preceding OpId from leader: term: 1 index: 46. (index mismatch)
I20240501 19:56:24.657033 10590 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 44, Time since last communication: 0.000s
I20240501 19:56:24.661418 9986 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } } }
I20240501 19:56:24.662928 9101 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Committing config change with OpId 1.46: config changed from index -1 to 46, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New config: { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } } }
I20240501 19:56:24.664785 8693 catalog_manager.cc:5484] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index -1 to 46, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:24.685556 8838 consensus_queue.cc:237] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:24.687580 9101 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240501 19:56:24.688386 10738 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 48, Time since last communication: 0.000s
I20240501 19:56:24.690575 10738 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index -1 to 49, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
I20240501 19:56:24.691212 9098 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index -1 to 49, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
W20240501 19:56:24.693341 8772 consensus_peers.cc:485] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 -> Peer c9564f3b39b549c1bd995354fa135c5f (127.5.139.68:46413): Couldn't send request to peer c9564f3b39b549c1bd995354fa135c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df35daed2f72400086ccd0b8c47ecab1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.695374 8693 catalog_manager.cc:5484] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index -1 to 49, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
I20240501 19:56:24.744037 9101 consensus_queue.cc:237] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 52, Committed index: 52, Last appended: 1.52, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
W20240501 19:56:24.749672 9034 consensus_peers.cc:485] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 -> Peer 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483): Couldn't send request to peer 6977f9a52e9b4d06bcb12f9bee61553b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa477b2bd34d4b74b245314148e4851d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:24.750367 8970 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 52. Preceding OpId from leader: term: 1 index: 53. (index mismatch)
I20240501 19:56:24.753585 10642 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 53, Last known committed idx: 52, Time since last communication: 0.000s
I20240501 19:56:24.758584 10642 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.53: config changed from index -1 to 53, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:24.760825 8968 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Committing config change with OpId 1.53: config changed from index -1 to 53, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:24.761194 8693 catalog_manager.cc:5484] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index -1 to 53, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:24.785669 10807 ts_tablet_manager.cc:902] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: Initiating tablet copy from peer b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:24.786505 10807 tablet_copy_client.cc:308] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:35389
I20240501 19:56:24.796080 9121 tablet_copy_service.cc:140] P b3b45c9058d84b4e9ace42c9154ab0e0: Received BeginTabletCopySession request for tablet 58979b7e47734549bc48710ff1d2b1c1 from peer 6b4f42453a864314adc86aba2c3b3556 ({username='slave'} at 127.5.139.70:58801)
I20240501 19:56:24.796231 9121 tablet_copy_service.cc:161] P b3b45c9058d84b4e9ace42c9154ab0e0: Beginning new tablet copy session on tablet 58979b7e47734549bc48710ff1d2b1c1 from peer 6b4f42453a864314adc86aba2c3b3556 at {username='slave'} at 127.5.139.70:58801: session id = 6b4f42453a864314adc86aba2c3b3556-58979b7e47734549bc48710ff1d2b1c1
I20240501 19:56:24.797672 9121 tablet_copy_source_session.cc:205] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:24.803355 10807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58979b7e47734549bc48710ff1d2b1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.823832 10824 raft_consensus.cc:482] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:24.824007 10824 raft_consensus.cc:504] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:24.830237 10807 tablet_copy_client.cc:783] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:24.830770 10807 tablet_copy_client.cc:647] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:24.837671 10807 tablet_copy_client.cc:523] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:24.839273 10824 leader_election.cc:288] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:24.840538 10807 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: Bootstrap starting.
I20240501 19:56:24.851405 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e64c2bfdce4caf9c91618400c7f5d6" candidate_uuid: "14798909ff724444a8be03c45c41929a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982" is_pre_election: true
I20240501 19:56:24.851547 8839 raft_consensus.cc:2454] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14798909ff724444a8be03c45c41929a in term 1.
I20240501 19:56:24.852011 10447 leader_election.cc:302] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, 14798909ff724444a8be03c45c41929a; no voters:
I20240501 19:56:24.853539 10824 raft_consensus.cc:2781] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 19:56:24.853628 10824 raft_consensus.cc:482] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:24.853668 10824 raft_consensus.cc:3037] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:24.855755 10824 raft_consensus.cc:504] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:24.857184 10824 leader_election.cc:288] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 2 election: Requested vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:24.862116 8838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6e64c2bfdce4caf9c91618400c7f5d6" candidate_uuid: "14798909ff724444a8be03c45c41929a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
I20240501 19:56:24.862226 8838 raft_consensus.cc:3037] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:24.863273 8838 raft_consensus.cc:2454] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14798909ff724444a8be03c45c41929a in term 2.
I20240501 19:56:24.863848 10831 ts_tablet_manager.cc:902] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: Initiating tablet copy from peer b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:24.864737 10831 tablet_copy_client.cc:308] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:35389
I20240501 19:56:24.865553 10447 leader_election.cc:302] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, 14798909ff724444a8be03c45c41929a; no voters:
I20240501 19:56:24.865808 10824 raft_consensus.cc:2781] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:24.865967 10824 raft_consensus.cc:686] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 2 LEADER]: Becoming Leader. State: Replica: 14798909ff724444a8be03c45c41929a, State: Running, Role: LEADER
I20240501 19:56:24.866247 10824 consensus_queue.cc:237] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } }
I20240501 19:56:24.869289 10807 log.cc:824] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:24.869295 8694 catalog_manager.cc:5484] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "14798909ff724444a8be03c45c41929a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:24.873356 9121 tablet_copy_service.cc:140] P b3b45c9058d84b4e9ace42c9154ab0e0: Received BeginTabletCopySession request for tablet 74706b357b2b43549757d24fa9a9d282 from peer c9564f3b39b549c1bd995354fa135c5f ({username='slave'} at 127.5.139.68:59875)
I20240501 19:56:24.873450 9121 tablet_copy_service.cc:161] P b3b45c9058d84b4e9ace42c9154ab0e0: Beginning new tablet copy session on tablet 74706b357b2b43549757d24fa9a9d282 from peer c9564f3b39b549c1bd995354fa135c5f at {username='slave'} at 127.5.139.68:59875: session id = c9564f3b39b549c1bd995354fa135c5f-74706b357b2b43549757d24fa9a9d282
I20240501 19:56:24.874431 9121 tablet_copy_source_session.cc:205] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:24.877794 10831 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74706b357b2b43549757d24fa9a9d282. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.887485 10845 ts_tablet_manager.cc:902] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Initiating tablet copy from peer 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:24.890776 10845 tablet_copy_client.cc:308] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38731
I20240501 19:56:24.893354 10831 tablet_copy_client.cc:783] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:24.893615 10831 tablet_copy_client.cc:647] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:24.895265 10831 tablet_copy_client.cc:523] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:24.897601 10831 tablet_bootstrap.cc:492] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap starting.
I20240501 19:56:24.912501 10831 log.cc:824] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:24.949869 10863 ts_tablet_manager.cc:902] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: Initiating tablet copy from peer b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:24.962992 10863 tablet_copy_client.cc:308] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:35389
I20240501 19:56:24.966225 10807 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: Bootstrap replayed 1/1 log segments. Stats: ops{read=56 overwritten=0 applied=56 ignored=0} inserts{seen=687 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:24.966703 10807 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: Bootstrap complete.
I20240501 19:56:24.967041 10807 ts_tablet_manager.cc:1367] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: Time spent bootstrapping tablet: real 0.127s user 0.062s sys 0.008s
I20240501 19:56:24.967826 9121 tablet_copy_service.cc:140] P b3b45c9058d84b4e9ace42c9154ab0e0: Received BeginTabletCopySession request for tablet 0f1ab4dc3bb841c2b352cee4ec7aa373 from peer c9564f3b39b549c1bd995354fa135c5f ({username='slave'} at 127.5.139.68:59875)
I20240501 19:56:24.967914 9121 tablet_copy_service.cc:161] P b3b45c9058d84b4e9ace42c9154ab0e0: Beginning new tablet copy session on tablet 0f1ab4dc3bb841c2b352cee4ec7aa373 from peer c9564f3b39b549c1bd995354fa135c5f at {username='slave'} at 127.5.139.68:59875: session id = c9564f3b39b549c1bd995354fa135c5f-0f1ab4dc3bb841c2b352cee4ec7aa373
I20240501 19:56:24.968751 9121 tablet_copy_source_session.cc:205] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:24.969659 10807 raft_consensus.cc:348] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:24.969910 10807 raft_consensus.cc:729] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b4f42453a864314adc86aba2c3b3556, State: Initialized, Role: LEARNER
I20240501 19:56:24.970253 10807 consensus_queue.cc:260] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 56, Last appended: 1.56, Last appended by leader: 56, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 42 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:24.971112 10807 ts_tablet_manager.cc:1398] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 19:56:24.971277 10863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f1ab4dc3bb841c2b352cee4ec7aa373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:24.973553 10863 tablet_copy_client.cc:783] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:24.973695 10863 tablet_copy_client.cc:647] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:24.977363 9121 tablet_copy_service.cc:342] P b3b45c9058d84b4e9ace42c9154ab0e0: Request end of tablet copy session 6b4f42453a864314adc86aba2c3b3556-58979b7e47734549bc48710ff1d2b1c1 received from {username='slave'} at 127.5.139.70:58801
I20240501 19:56:24.977447 9121 tablet_copy_service.cc:433] P b3b45c9058d84b4e9ace42c9154ab0e0: ending tablet copy session 6b4f42453a864314adc86aba2c3b3556-58979b7e47734549bc48710ff1d2b1c1 on tablet 58979b7e47734549bc48710ff1d2b1c1 with peer 6b4f42453a864314adc86aba2c3b3556
I20240501 19:56:24.978435 10870 ts_tablet_manager.cc:902] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: Initiating tablet copy from peer ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:24.979550 10831 tablet_bootstrap.cc:492] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=53 overwritten=0 applied=53 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:24.979897 10831 tablet_bootstrap.cc:492] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap complete.
I20240501 19:56:24.980202 10831 ts_tablet_manager.cc:1367] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: Time spent bootstrapping tablet: real 0.083s user 0.058s sys 0.010s
I20240501 19:56:24.982585 10831 raft_consensus.cc:348] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:24.982808 10831 raft_consensus.cc:729] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9564f3b39b549c1bd995354fa135c5f, State: Initialized, Role: LEARNER
I20240501 19:56:24.983141 10831 consensus_queue.cc:260] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 53, Last appended: 1.53, Last appended by leader: 53, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 28 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:24.984066 10831 ts_tablet_manager.cc:1398] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240501 19:56:24.984952 9120 tablet_copy_service.cc:342] P b3b45c9058d84b4e9ace42c9154ab0e0: Request end of tablet copy session c9564f3b39b549c1bd995354fa135c5f-74706b357b2b43549757d24fa9a9d282 received from {username='slave'} at 127.5.139.68:59875
I20240501 19:56:24.985014 9120 tablet_copy_service.cc:433] P b3b45c9058d84b4e9ace42c9154ab0e0: ending tablet copy session c9564f3b39b549c1bd995354fa135c5f-74706b357b2b43549757d24fa9a9d282 on tablet 74706b357b2b43549757d24fa9a9d282 with peer c9564f3b39b549c1bd995354fa135c5f
I20240501 19:56:24.986420 10863 tablet_copy_client.cc:523] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:24.987684 10870 tablet_copy_client.cc:308] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:43725
I20240501 19:56:24.987700 10863 tablet_bootstrap.cc:492] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap starting.
I20240501 19:56:24.993253 8859 tablet_copy_service.cc:140] P 0f2b49c6d3dc49fabee40802e73ac982: Received BeginTabletCopySession request for tablet d827d0b0d5ba4969908768c96f92c1f3 from peer 6977f9a52e9b4d06bcb12f9bee61553b ({username='slave'} at 127.5.139.69:52827)
I20240501 19:56:24.993386 8859 tablet_copy_service.cc:161] P 0f2b49c6d3dc49fabee40802e73ac982: Beginning new tablet copy session on tablet d827d0b0d5ba4969908768c96f92c1f3 from peer 6977f9a52e9b4d06bcb12f9bee61553b at {username='slave'} at 127.5.139.69:52827: session id = 6977f9a52e9b4d06bcb12f9bee61553b-d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:24.994735 8859 tablet_copy_source_session.cc:205] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:24.996280 8990 tablet_copy_service.cc:140] P ac13bdb319c448dba072fb9e62d6893b: Received BeginTabletCopySession request for tablet 0ca216ce0c6a420f8c90273a02ad8cf4 from peer 14798909ff724444a8be03c45c41929a ({username='slave'} at 127.5.139.71:48051)
I20240501 19:56:24.996412 8990 tablet_copy_service.cc:161] P ac13bdb319c448dba072fb9e62d6893b: Beginning new tablet copy session on tablet 0ca216ce0c6a420f8c90273a02ad8cf4 from peer 14798909ff724444a8be03c45c41929a at {username='slave'} at 127.5.139.71:48051: session id = 14798909ff724444a8be03c45c41929a-0ca216ce0c6a420f8c90273a02ad8cf4
I20240501 19:56:24.997691 8990 tablet_copy_source_session.cc:205] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:24.999481 10845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d827d0b0d5ba4969908768c96f92c1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.012073 10845 tablet_copy_client.cc:783] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:25.012318 10845 tablet_copy_client.cc:647] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:25.013437 10870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ca216ce0c6a420f8c90273a02ad8cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.014737 10845 tablet_copy_client.cc:523] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:25.016722 10870 tablet_copy_client.cc:783] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:25.016906 10870 tablet_copy_client.cc:647] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:25.027396 10870 tablet_copy_client.cc:523] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:25.028913 10870 tablet_bootstrap.cc:492] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: Bootstrap starting.
I20240501 19:56:25.033742 10807 ts_tablet_manager.cc:902] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: Initiating tablet copy from peer b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:25.034229 10807 tablet_copy_client.cc:308] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:35389
I20240501 19:56:25.034679 9121 tablet_copy_service.cc:140] P b3b45c9058d84b4e9ace42c9154ab0e0: Received BeginTabletCopySession request for tablet c869ad944b7a45d39ec171b0add850fb from peer 6b4f42453a864314adc86aba2c3b3556 ({username='slave'} at 127.5.139.70:58801)
I20240501 19:56:25.034787 9121 tablet_copy_service.cc:161] P b3b45c9058d84b4e9ace42c9154ab0e0: Beginning new tablet copy session on tablet c869ad944b7a45d39ec171b0add850fb from peer 6b4f42453a864314adc86aba2c3b3556 at {username='slave'} at 127.5.139.70:58801: session id = 6b4f42453a864314adc86aba2c3b3556-c869ad944b7a45d39ec171b0add850fb
I20240501 19:56:25.035686 9121 tablet_copy_source_session.cc:205] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:25.037181 10807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c869ad944b7a45d39ec171b0add850fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.037801 10845 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap starting.
I20240501 19:56:25.040369 10807 tablet_copy_client.cc:783] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:25.040515 10807 tablet_copy_client.cc:647] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:25.046682 10807 tablet_copy_client.cc:523] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:25.047935 10807 tablet_bootstrap.cc:492] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: Bootstrap starting.
I20240501 19:56:25.050633 10845 log.cc:824] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:25.065112 10889 ts_tablet_manager.cc:902] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: Initiating tablet copy from peer b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:25.065760 10889 tablet_copy_client.cc:308] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:35389
I20240501 19:56:25.070492 9121 tablet_copy_service.cc:140] P b3b45c9058d84b4e9ace42c9154ab0e0: Received BeginTabletCopySession request for tablet 49f4147edb7349bcbf51994e59501d19 from peer 14798909ff724444a8be03c45c41929a ({username='slave'} at 127.5.139.71:46405)
I20240501 19:56:25.070587 9121 tablet_copy_service.cc:161] P b3b45c9058d84b4e9ace42c9154ab0e0: Beginning new tablet copy session on tablet 49f4147edb7349bcbf51994e59501d19 from peer 14798909ff724444a8be03c45c41929a at {username='slave'} at 127.5.139.71:46405: session id = 14798909ff724444a8be03c45c41929a-49f4147edb7349bcbf51994e59501d19
I20240501 19:56:25.071460 9121 tablet_copy_source_session.cc:205] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:25.080714 10889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f4147edb7349bcbf51994e59501d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.083613 10889 tablet_copy_client.cc:783] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:25.083815 10889 tablet_copy_client.cc:647] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:25.086191 10889 tablet_copy_client.cc:523] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:25.087384 10889 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: Bootstrap starting.
I20240501 19:56:25.108095 10863 tablet_bootstrap.cc:492] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=42 overwritten=0 applied=41 ignored=0} inserts{seen=672 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:25.108709 10863 tablet_bootstrap.cc:492] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap complete.
I20240501 19:56:25.108909 10863 ts_tablet_manager.cc:1367] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: Time spent bootstrapping tablet: real 0.121s user 0.047s sys 0.007s
I20240501 19:56:25.109459 10863 raft_consensus.cc:348] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:25.110921 10863 raft_consensus.cc:729] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9564f3b39b549c1bd995354fa135c5f, State: Initialized, Role: LEARNER
I20240501 19:56:25.111052 10863 consensus_queue.cc:260] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 41, Last appended: 1.42, Last appended by leader: 42, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 26 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:25.111318 10863 ts_tablet_manager.cc:1398] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:25.113772 9121 tablet_copy_service.cc:342] P b3b45c9058d84b4e9ace42c9154ab0e0: Request end of tablet copy session c9564f3b39b549c1bd995354fa135c5f-0f1ab4dc3bb841c2b352cee4ec7aa373 received from {username='slave'} at 127.5.139.68:59875
I20240501 19:56:25.113853 9121 tablet_copy_service.cc:433] P b3b45c9058d84b4e9ace42c9154ab0e0: ending tablet copy session c9564f3b39b549c1bd995354fa135c5f-0f1ab4dc3bb841c2b352cee4ec7aa373 on tablet 0f1ab4dc3bb841c2b352cee4ec7aa373 with peer c9564f3b39b549c1bd995354fa135c5f
I20240501 19:56:25.169800 10807 tablet_bootstrap.cc:492] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=70 ignored=0} inserts{seen=701 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:25.170068 10807 tablet_bootstrap.cc:492] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: Bootstrap complete.
I20240501 19:56:25.170266 10807 ts_tablet_manager.cc:1367] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: Time spent bootstrapping tablet: real 0.122s user 0.071s sys 0.004s
I20240501 19:56:25.170771 10807 raft_consensus.cc:348] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:25.170886 10807 raft_consensus.cc:729] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b4f42453a864314adc86aba2c3b3556, State: Initialized, Role: LEARNER
I20240501 19:56:25.170974 10807 consensus_queue.cc:260] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 70, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:25.171273 10807 ts_tablet_manager.cc:1398] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.171707 9121 tablet_copy_service.cc:342] P b3b45c9058d84b4e9ace42c9154ab0e0: Request end of tablet copy session 6b4f42453a864314adc86aba2c3b3556-c869ad944b7a45d39ec171b0add850fb received from {username='slave'} at 127.5.139.70:58801
I20240501 19:56:25.171787 9121 tablet_copy_service.cc:433] P b3b45c9058d84b4e9ace42c9154ab0e0: ending tablet copy session 6b4f42453a864314adc86aba2c3b3556-c869ad944b7a45d39ec171b0add850fb on tablet c869ad944b7a45d39ec171b0add850fb with peer 6b4f42453a864314adc86aba2c3b3556
I20240501 19:56:25.190253 10383 raft_consensus.cc:1203] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.41->[1.42-1.61] Dedup: 1.56->[1.57-1.61]
I20240501 19:56:25.197573 10870 tablet_bootstrap.cc:492] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: Bootstrap replayed 1/1 log segments. Stats: ops{read=38 overwritten=0 applied=38 ignored=0} inserts{seen=723 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:25.198009 10870 tablet_bootstrap.cc:492] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: Bootstrap complete.
I20240501 19:56:25.198259 10870 ts_tablet_manager.cc:1367] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: Time spent bootstrapping tablet: real 0.169s user 0.058s sys 0.000s
I20240501 19:56:25.198944 10870 raft_consensus.cc:348] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } }
I20240501 19:56:25.199143 10870 raft_consensus.cc:729] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14798909ff724444a8be03c45c41929a, State: Initialized, Role: LEARNER
I20240501 19:56:25.199652 10870 consensus_queue.cc:260] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 38, Last appended: 1.38, Last appended by leader: 38, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 24 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } }
I20240501 19:56:25.201475 10870 ts_tablet_manager.cc:1398] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:25.204432 8990 tablet_copy_service.cc:342] P ac13bdb319c448dba072fb9e62d6893b: Request end of tablet copy session 14798909ff724444a8be03c45c41929a-0ca216ce0c6a420f8c90273a02ad8cf4 received from {username='slave'} at 127.5.139.71:48051
I20240501 19:56:25.204535 8990 tablet_copy_service.cc:433] P ac13bdb319c448dba072fb9e62d6893b: ending tablet copy session 14798909ff724444a8be03c45c41929a-0ca216ce0c6a420f8c90273a02ad8cf4 on tablet 0ca216ce0c6a420f8c90273a02ad8cf4 with peer 14798909ff724444a8be03c45c41929a
I20240501 19:56:25.205592 10901 ts_tablet_manager.cc:902] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: Initiating tablet copy from peer 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:25.206681 10901 tablet_copy_client.cc:308] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38731
I20240501 19:56:25.207141 8859 tablet_copy_service.cc:140] P 0f2b49c6d3dc49fabee40802e73ac982: Received BeginTabletCopySession request for tablet 0c4e313b03cc44d0a2f7864f7e9d4f90 from peer 6977f9a52e9b4d06bcb12f9bee61553b ({username='slave'} at 127.5.139.69:52827)
I20240501 19:56:25.207235 8859 tablet_copy_service.cc:161] P 0f2b49c6d3dc49fabee40802e73ac982: Beginning new tablet copy session on tablet 0c4e313b03cc44d0a2f7864f7e9d4f90 from peer 6977f9a52e9b4d06bcb12f9bee61553b at {username='slave'} at 127.5.139.69:52827: session id = 6977f9a52e9b4d06bcb12f9bee61553b-0c4e313b03cc44d0a2f7864f7e9d4f90
I20240501 19:56:25.208168 8859 tablet_copy_source_session.cc:205] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:25.209828 10901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4e313b03cc44d0a2f7864f7e9d4f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.211462 10901 tablet_copy_client.cc:783] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:25.211604 10901 tablet_copy_client.cc:647] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:25.213915 10901 tablet_copy_client.cc:523] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:25.215090 10901 tablet_bootstrap.cc:492] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap starting.
I20240501 19:56:25.216909 10845 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap replayed 1/1 log segments. Stats: ops{read=65 overwritten=0 applied=65 ignored=0} inserts{seen=737 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:25.217257 10845 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap complete.
I20240501 19:56:25.217587 10845 ts_tablet_manager.cc:1367] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent bootstrapping tablet: real 0.180s user 0.074s sys 0.011s
I20240501 19:56:25.226353 10845 raft_consensus.cc:348] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:25.226620 10845 raft_consensus.cc:729] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6977f9a52e9b4d06bcb12f9bee61553b, State: Initialized, Role: LEARNER
I20240501 19:56:25.226924 10646 raft_consensus.cc:1050] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: attempting to promote NON_VOTER 6b4f42453a864314adc86aba2c3b3556 to VOTER
I20240501 19:56:25.226979 10845 consensus_queue.cc:260] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 45 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:25.227312 10646 consensus_queue.cc:237] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 62, Committed index: 62, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:25.229709 8839 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240501 19:56:25.230209 10802 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240501 19:56:25.231215 10383 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240501 19:56:25.231294 10845 ts_tablet_manager.cc:1398] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent starting tablet: real 0.014s user 0.004s sys 0.000s
I20240501 19:56:25.232148 10802 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 62, Time since last communication: 0.000s
I20240501 19:56:25.234133 10802 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index 42 to 63, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.236667 8693 catalog_manager.cc:5484] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 42 to 63, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.236505 10382 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index 42 to 63, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.245980 8859 tablet_copy_service.cc:342] P 0f2b49c6d3dc49fabee40802e73ac982: Request end of tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-d827d0b0d5ba4969908768c96f92c1f3 received from {username='slave'} at 127.5.139.69:52827
I20240501 19:56:25.246086 8859 tablet_copy_service.cc:433] P 0f2b49c6d3dc49fabee40802e73ac982: ending tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-d827d0b0d5ba4969908768c96f92c1f3 on tablet d827d0b0d5ba4969908768c96f92c1f3 with peer 6977f9a52e9b4d06bcb12f9bee61553b
I20240501 19:56:25.253440 10807 ts_tablet_manager.cc:902] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: Initiating tablet copy from peer ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:25.253998 10807 tablet_copy_client.cc:308] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:43725
I20240501 19:56:25.258543 8990 tablet_copy_service.cc:140] P ac13bdb319c448dba072fb9e62d6893b: Received BeginTabletCopySession request for tablet 68c8311312bd4b50bdbedf414ada6bd2 from peer 6b4f42453a864314adc86aba2c3b3556 ({username='slave'} at 127.5.139.70:54631)
I20240501 19:56:25.258651 8990 tablet_copy_service.cc:161] P ac13bdb319c448dba072fb9e62d6893b: Beginning new tablet copy session on tablet 68c8311312bd4b50bdbedf414ada6bd2 from peer 6b4f42453a864314adc86aba2c3b3556 at {username='slave'} at 127.5.139.70:54631: session id = 6b4f42453a864314adc86aba2c3b3556-68c8311312bd4b50bdbedf414ada6bd2
I20240501 19:56:25.259586 8990 tablet_copy_source_session.cc:205] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:25.260583 10807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c8311312bd4b50bdbedf414ada6bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.261966 8837 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index 42 to 63, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.262722 10807 tablet_copy_client.cc:783] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:25.262874 10807 tablet_copy_client.cc:647] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:25.280359 10889 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: Bootstrap replayed 1/1 log segments. Stats: ops{read=77 overwritten=0 applied=77 ignored=0} inserts{seen=762 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:25.280745 10889 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: Bootstrap complete.
I20240501 19:56:25.280953 10889 ts_tablet_manager.cc:1367] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: Time spent bootstrapping tablet: real 0.194s user 0.078s sys 0.004s
I20240501 19:56:25.281472 10889 raft_consensus.cc:348] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } }
I20240501 19:56:25.281612 10889 raft_consensus.cc:729] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14798909ff724444a8be03c45c41929a, State: Initialized, Role: LEARNER
I20240501 19:56:25.281730 10889 consensus_queue.cc:260] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 77, Last appended: 1.77, Last appended by leader: 77, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 59 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } }
I20240501 19:56:25.282122 10889 ts_tablet_manager.cc:1398] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:25.282519 9121 tablet_copy_service.cc:342] P b3b45c9058d84b4e9ace42c9154ab0e0: Request end of tablet copy session 14798909ff724444a8be03c45c41929a-49f4147edb7349bcbf51994e59501d19 received from {username='slave'} at 127.5.139.71:46405
I20240501 19:56:25.282596 9121 tablet_copy_service.cc:433] P b3b45c9058d84b4e9ace42c9154ab0e0: ending tablet copy session 14798909ff724444a8be03c45c41929a-49f4147edb7349bcbf51994e59501d19 on tablet 49f4147edb7349bcbf51994e59501d19 with peer 14798909ff724444a8be03c45c41929a
I20240501 19:56:25.285439 9100 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58979b7e47734549bc48710ff1d2b1c1"
dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45134
I20240501 19:56:25.285622 9100 raft_consensus.cc:595] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:25.288386 10863 ts_tablet_manager.cc:902] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: Initiating tablet copy from peer 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:25.288898 10863 tablet_copy_client.cc:308] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38731
I20240501 19:56:25.291364 10807 tablet_copy_client.cc:523] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:25.292701 10807 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: Bootstrap starting.
I20240501 19:56:25.295269 8859 tablet_copy_service.cc:140] P 0f2b49c6d3dc49fabee40802e73ac982: Received BeginTabletCopySession request for tablet df35daed2f72400086ccd0b8c47ecab1 from peer c9564f3b39b549c1bd995354fa135c5f ({username='slave'} at 127.5.139.68:56291)
I20240501 19:56:25.295372 8859 tablet_copy_service.cc:161] P 0f2b49c6d3dc49fabee40802e73ac982: Beginning new tablet copy session on tablet df35daed2f72400086ccd0b8c47ecab1 from peer c9564f3b39b549c1bd995354fa135c5f at {username='slave'} at 127.5.139.68:56291: session id = c9564f3b39b549c1bd995354fa135c5f-df35daed2f72400086ccd0b8c47ecab1
I20240501 19:56:25.296293 8859 tablet_copy_source_session.cc:205] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:25.298138 10863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df35daed2f72400086ccd0b8c47ecab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:25.300384 10863 tablet_copy_client.cc:783] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:25.300534 10863 tablet_copy_client.cc:647] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:25.304407 10121 raft_consensus.cc:1203] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.27->[1.28-1.63] Dedup: 1.53->[1.54-1.63]
I20240501 19:56:25.309031 10845 ts_tablet_manager.cc:902] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: Initiating tablet copy from peer b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:25.309577 10845 tablet_copy_client.cc:308] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:35389
I20240501 19:56:25.313270 10863 tablet_copy_client.cc:523] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 19:56:25.314272 9060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.315425 10863 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap starting.
W20240501 19:56:25.317706 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.319286 9121 tablet_copy_service.cc:140] P b3b45c9058d84b4e9ace42c9154ab0e0: Received BeginTabletCopySession request for tablet aa477b2bd34d4b74b245314148e4851d from peer 6977f9a52e9b4d06bcb12f9bee61553b ({username='slave'} at 127.5.139.69:33027)
I20240501 19:56:25.319382 9121 tablet_copy_service.cc:161] P b3b45c9058d84b4e9ace42c9154ab0e0: Beginning new tablet copy session on tablet aa477b2bd34d4b74b245314148e4851d from peer 6977f9a52e9b4d06bcb12f9bee61553b at {username='slave'} at 127.5.139.69:33027: session id = 6977f9a52e9b4d06bcb12f9bee61553b-aa477b2bd34d4b74b245314148e4851d
I20240501 19:56:25.320235 9121 tablet_copy_source_session.cc:205] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 19:56:25.326004 9060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.328347 10845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa477b2bd34d4b74b245314148e4851d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 19:56:25.332149 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
W20240501 19:56:25.339351 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.347349 10845 tablet_copy_client.cc:783] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:25.347520 10845 tablet_copy_client.cc:647] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:25.356231 10845 tablet_copy_client.cc:523] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 19:56:25.356395 9058 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.357522 10845 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap starting.
W20240501 19:56:25.363531 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
W20240501 19:56:25.382566 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.385963 10510 raft_consensus.cc:1203] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.23->[1.24-1.47] Dedup: 1.38->[1.39-1.47]
W20240501 19:56:25.398087 9058 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.398119 10121 raft_consensus.cc:1203] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.25->[1.26-1.52] Dedup: 1.42->[1.43-1.52]
I20240501 19:56:25.413189 10383 raft_consensus.cc:1203] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.45->[1.46-1.80] Dedup: 1.70->[1.71-1.80]
I20240501 19:56:25.424206 10802 raft_consensus.cc:1050] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0: attempting to promote NON_VOTER c9564f3b39b549c1bd995354fa135c5f to VOTER
I20240501 19:56:25.424549 10802 consensus_queue.cc:237] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 53, Committed index: 53, Last appended: 1.53, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:25.428282 10121 raft_consensus.cc:1261] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240501 19:56:25.428856 10646 consensus_queue.cc:1035] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 54, Last known committed idx: 53, Time since last communication: 0.000s
I20240501 19:56:25.431162 10646 raft_consensus.cc:2932] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.54: config changed from index 26 to 54, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.431615 10121 raft_consensus.cc:2932] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Committing config change with OpId 1.54: config changed from index 26 to 54, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.433496 9986 raft_consensus.cc:1050] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b: attempting to promote NON_VOTER 14798909ff724444a8be03c45c41929a to VOTER
I20240501 19:56:25.433928 9986 consensus_queue.cc:237] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 47, Committed index: 47, Last appended: 1.48, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } }
I20240501 19:56:25.434125 8694 catalog_manager.cc:5484] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 26 to 54, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 54 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.435350 10510 raft_consensus.cc:1261] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240501 19:56:25.435823 10590 consensus_queue.cc:1035] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 47, Time since last communication: 0.000s
I20240501 19:56:25.437150 9100 consensus_queue.cc:237] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 54, Committed index: 54, Last appended: 1.54, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:25.437717 9098 raft_consensus.cc:1261] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 49. (index mismatch)
I20240501 19:56:25.438256 10590 consensus_queue.cc:1035] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 49, Last known committed idx: 47, Time since last communication: 0.000s
I20240501 19:56:25.438302 10121 raft_consensus.cc:1261] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 54. Preceding OpId from leader: term: 1 index: 55. (index mismatch)
W20240501 19:56:25.438401 9060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.439456 10642 consensus_queue.cc:1035] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 55, Last known committed idx: 54, Time since last communication: 0.000s
I20240501 19:56:25.440615 10590 raft_consensus.cc:2932] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.49: config changed from index 24 to 49, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:25.444305 10252 raft_consensus.cc:1203] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.44->[1.45-1.70] Dedup: 1.65->[1.66-1.70]
I20240501 19:56:25.447594 10901 tablet_bootstrap.cc:492] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap replayed 1/1 log segments. Stats: ops{read=72 overwritten=0 applied=71 ignored=0} inserts{seen=743 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:25.448021 10901 tablet_bootstrap.cc:492] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap complete.
I20240501 19:56:25.448253 10901 ts_tablet_manager.cc:1367] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent bootstrapping tablet: real 0.233s user 0.074s sys 0.000s
I20240501 19:56:25.448935 10901 raft_consensus.cc:348] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:25.449342 10901 raft_consensus.cc:729] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6977f9a52e9b4d06bcb12f9bee61553b, State: Initialized, Role: LEARNER
I20240501 19:56:25.449419 8694 catalog_manager.cc:5484] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index 24 to 49, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:25.449501 10901 consensus_queue.cc:260] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 71, Last appended: 1.72, Last appended by leader: 72, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:25.449947 10901 ts_tablet_manager.cc:1398] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:25.450542 8859 tablet_copy_service.cc:342] P 0f2b49c6d3dc49fabee40802e73ac982: Request end of tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-0c4e313b03cc44d0a2f7864f7e9d4f90 received from {username='slave'} at 127.5.139.69:52827
I20240501 19:56:25.450641 8859 tablet_copy_service.cc:433] P 0f2b49c6d3dc49fabee40802e73ac982: ending tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-0c4e313b03cc44d0a2f7864f7e9d4f90 on tablet 0c4e313b03cc44d0a2f7864f7e9d4f90 with peer 6977f9a52e9b4d06bcb12f9bee61553b
I20240501 19:56:25.452859 10802 raft_consensus.cc:2932] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.55: config changed from index 54 to 55, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.452968 9098 raft_consensus.cc:2932] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 24 to 49, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
W20240501 19:56:25.453819 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.456490 10510 raft_consensus.cc:2932] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Committing config change with OpId 1.49: config changed from index 24 to 49, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:25.456802 8684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f1ab4dc3bb841c2b352cee4ec7aa373 with cas_config_opid_index 54: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.457285 8693 catalog_manager.cc:5484] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 54 to 55, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.457479 10121 raft_consensus.cc:2932] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Committing config change with OpId 1.55: config changed from index 54 to 55, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 55 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.458031 10807 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=64 ignored=0} inserts{seen=695 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:25.458278 10807 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: Bootstrap complete.
I20240501 19:56:25.458465 10807 ts_tablet_manager.cc:1367] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: Time spent bootstrapping tablet: real 0.166s user 0.061s sys 0.004s
I20240501 19:56:25.458943 10807 raft_consensus.cc:348] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:25.459071 10807 raft_consensus.cc:729] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b4f42453a864314adc86aba2c3b3556, State: Initialized, Role: LEARNER
I20240501 19:56:25.459162 10807 consensus_queue.cc:260] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 46 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:25.459405 10807 ts_tablet_manager.cc:1398] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:25.459909 8969 raft_consensus.cc:1261] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 53. Preceding OpId from leader: term: 1 index: 54. (index mismatch)
I20240501 19:56:25.460559 8990 tablet_copy_service.cc:342] P ac13bdb319c448dba072fb9e62d6893b: Request end of tablet copy session 6b4f42453a864314adc86aba2c3b3556-68c8311312bd4b50bdbedf414ada6bd2 received from {username='slave'} at 127.5.139.70:54631
I20240501 19:56:25.460624 8990 tablet_copy_service.cc:433] P ac13bdb319c448dba072fb9e62d6893b: ending tablet copy session 6b4f42453a864314adc86aba2c3b3556-68c8311312bd4b50bdbedf414ada6bd2 on tablet 68c8311312bd4b50bdbedf414ada6bd2 with peer 6b4f42453a864314adc86aba2c3b3556
I20240501 19:56:25.465173 8949 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f1ab4dc3bb841c2b352cee4ec7aa373 with delete_type TABLET_DATA_TOMBSTONED (TS ac13bdb319c448dba072fb9e62d6893b not found in new config with opid_index 55) from {username='slave'} at 127.0.0.1:45626
I20240501 19:56:25.465608 10964 tablet_replica.cc:330] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b: stopping tablet replica
I20240501 19:56:25.465844 8969 consensus_queue.cc:237] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 49, Committed index: 49, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } }
I20240501 19:56:25.466161 10964 raft_consensus.cc:2229] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 19:56:25.466291 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.466562 10964 raft_consensus.cc:2258] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.467690 10964 ts_tablet_manager.cc:1875] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.470172 10964 ts_tablet_manager.cc:1888] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.53
I20240501 19:56:25.470311 10964 log.cc:1196] T 0f1ab4dc3bb841c2b352cee4ec7aa373 P ac13bdb319c448dba072fb9e62d6893b: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/0f1ab4dc3bb841c2b352cee4ec7aa373
I20240501 19:56:25.470965 8682 catalog_manager.cc:4830] TS ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725): tablet 0f1ab4dc3bb841c2b352cee4ec7aa373 (table rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]) successfully deleted
I20240501 19:56:25.472255 10510 raft_consensus.cc:1261] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 49. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240501 19:56:25.474448 10616 consensus_queue.cc:1035] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 49, Time since last communication: 0.000s
I20240501 19:56:25.475208 8837 raft_consensus.cc:1261] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 19:56:25.475706 10837 consensus_queue.cc:1035] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:25.476198 10642 raft_consensus.cc:1050] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0: attempting to promote NON_VOTER 6b4f42453a864314adc86aba2c3b3556 to VOTER
I20240501 19:56:25.476514 10642 consensus_queue.cc:237] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:25.477596 10383 raft_consensus.cc:1261] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240501 19:56:25.477656 8837 raft_consensus.cc:1261] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240501 19:56:25.477983 10802 consensus_queue.cc:1035] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 19:56:25.479583 10802 consensus_queue.cc:1035] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 19:56:25.479630 10738 raft_consensus.cc:1050] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: attempting to promote NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b to VOTER
I20240501 19:56:25.479946 10738 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.72, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:25.481911 10251 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 72. Preceding OpId from leader: term: 1 index: 73. (index mismatch)
I20240501 19:56:25.482075 10802 raft_consensus.cc:2932] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 46 to 82, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.482503 10653 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 73, Last known committed idx: 71, Time since last communication: 0.000s
W20240501 19:56:25.484580 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.484962 10653 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.73: config changed from index 45 to 73, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.484982 8696 catalog_manager.cc:5484] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 46 to 82, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.487263 10383 raft_consensus.cc:2932] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 46 to 82, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.488137 10252 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.73: config changed from index 45 to 73, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.488454 8837 raft_consensus.cc:2932] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 46 to 82, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.489899 8691 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index 45 to 73, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 73 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.491297 9100 consensus_queue.cc:237] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:25.493171 10382 raft_consensus.cc:1261] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240501 19:56:25.494514 10802 consensus_queue.cc:1035] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 82, Time since last communication: 0.000s
I20240501 19:56:25.496284 10630 raft_consensus.cc:2932] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.496867 8837 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 73, Committed index: 73, Last appended: 1.73, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:25.499055 10382 raft_consensus.cc:2932] T c869ad944b7a45d39ec171b0add850fb P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.499156 8684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c869ad944b7a45d39ec171b0add850fb with cas_config_opid_index 82: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.499850 10251 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 73. Preceding OpId from leader: term: 1 index: 74. (index mismatch)
I20240501 19:56:25.500432 9923 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 74, Last known committed idx: 73, Time since last communication: 0.000s
I20240501 19:56:25.500564 8693 catalog_manager.cc:5484] T c869ad944b7a45d39ec171b0add850fb P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 82 to 84, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:25.502076 9923 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.74: config changed from index 73 to 74, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
W20240501 19:56:25.502646 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.504814 8691 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index 73 to 74, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.507421 8681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d827d0b0d5ba4969908768c96f92c1f3 with cas_config_opid_index 73: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.507699 8819 tablet_service.cc:1514] Processing DeleteTablet for tablet c869ad944b7a45d39ec171b0add850fb with delete_type TABLET_DATA_TOMBSTONED (TS 0f2b49c6d3dc49fabee40802e73ac982 not found in new config with opid_index 84) from {username='slave'} at 127.0.0.1:48934
I20240501 19:56:25.507841 10251 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.74: config changed from index 73 to 74, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 74 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.508985 8949 tablet_service.cc:1514] Processing DeleteTablet for tablet d827d0b0d5ba4969908768c96f92c1f3 with delete_type TABLET_DATA_TOMBSTONED (TS ac13bdb319c448dba072fb9e62d6893b not found in new config with opid_index 74) from {username='slave'} at 127.0.0.1:45626
I20240501 19:56:25.509577 10964 tablet_replica.cc:330] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b: stopping tablet replica
I20240501 19:56:25.510396 10964 raft_consensus.cc:2229] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.510736 10964 raft_consensus.cc:2258] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.511536 10964 ts_tablet_manager.cc:1875] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.513859 10964 ts_tablet_manager.cc:1888] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.71
I20240501 19:56:25.513981 10964 log.cc:1196] T d827d0b0d5ba4969908768c96f92c1f3 P ac13bdb319c448dba072fb9e62d6893b: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:25.517252 10966 tablet_replica.cc:330] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: stopping tablet replica
I20240501 19:56:25.517609 10966 raft_consensus.cc:2229] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.517807 10966 pending_rounds.cc:62] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: Trying to abort 1 pending ops.
I20240501 19:56:25.517910 10966 pending_rounds.cc:68] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: Aborting op as it isn't in flight: id { term: 1 index: 83 } timestamp: 7022974506942672896 op_type: WRITE_OP write_request { tablet_id: "c869ad944b7a45d39ec171b0add850fb" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000@\036\362u\"\245\314<\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""263e3fe481a98c5a" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022974506889539584 authz_token { token_data: "\010\223\277\312\261\006\"3\n\005slave\022*\n 3c90b69a201147fa8697594a2940e3a2\020\001\030\001 \001(\001" signature: "<redacted>""\022\244\0312\373Z\226*\020\222/\315\262\232\0244\315\276^1\\h\266;t\340xR\024\351\037\335]\205\346|EJ\254ev\335\340\327\211\017a~\350\035\376\353\311y\365BG\017\355\"\330\302\251\267" signing_key_seq_num: 0 } } request_id { client_id: "d64764e15e6c43cb8a5c3b0f7d1181c4" seq_no: 208 first_incomplete_seq_no: 208 attempt_no: 0 }
I20240501 19:56:25.518532 10966 raft_consensus.cc:2258] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 19:56:25.519268 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.519744 10966 ts_tablet_manager.cc:1875] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.521323 8682 catalog_manager.cc:4830] TS ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725): tablet d827d0b0d5ba4969908768c96f92c1f3 (table rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]) successfully deleted
I20240501 19:56:25.522147 10966 ts_tablet_manager.cc:1888] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.83
I20240501 19:56:25.523348 9986 raft_consensus.cc:2932] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index 49 to 50, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:25.523764 10510 raft_consensus.cc:2932] T 0ca216ce0c6a420f8c90273a02ad8cf4 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index 49 to 50, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:25.524037 10966 log.cc:1196] T c869ad944b7a45d39ec171b0add850fb P 0f2b49c6d3dc49fabee40802e73ac982: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/c869ad944b7a45d39ec171b0add850fb
I20240501 19:56:25.524855 8682 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ca216ce0c6a420f8c90273a02ad8cf4 with cas_config_opid_index 49: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.525305 8681 catalog_manager.cc:4830] TS 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731): tablet c869ad944b7a45d39ec171b0add850fb (table rebalance_test_table_1 [id=3c90b69a201147fa8697594a2940e3a2]) successfully deleted
I20240501 19:56:25.526093 8696 catalog_manager.cc:5484] T 0ca216ce0c6a420f8c90273a02ad8cf4 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index 49 to 50, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.528525 9080 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ca216ce0c6a420f8c90273a02ad8cf4 with delete_type TABLET_DATA_TOMBSTONED (TS b3b45c9058d84b4e9ace42c9154ab0e0 not found in new config with opid_index 50) from {username='slave'} at 127.0.0.1:44930
I20240501 19:56:25.537451 10967 tablet_replica.cc:330] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0: stopping tablet replica
W20240501 19:56:25.537693 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.537915 10967 raft_consensus.cc:2229] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.538151 10967 raft_consensus.cc:2258] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.548771 10967 ts_tablet_manager.cc:1875] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 19:56:25.557196 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.565979 10252 raft_consensus.cc:1203] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.48->[1.49-1.79] Dedup: 1.72->[1.73-1.79]
I20240501 19:56:25.571785 10967 ts_tablet_manager.cc:1888] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.49
I20240501 19:56:25.571894 10967 log.cc:1196] T 0ca216ce0c6a420f8c90273a02ad8cf4 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/0ca216ce0c6a420f8c90273a02ad8cf4
I20240501 19:56:25.572508 8684 catalog_manager.cc:4830] TS b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389): tablet 0ca216ce0c6a420f8c90273a02ad8cf4 (table rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]) successfully deleted
W20240501 19:56:25.579645 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.587127 10510 raft_consensus.cc:1203] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.58->[1.59-1.84] Dedup: 1.77->[1.78-1.84]
W20240501 19:56:25.601006 9060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.617506 10863 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=70 overwritten=0 applied=69 ignored=0} inserts{seen=754 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:25.617897 10863 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap complete.
I20240501 19:56:25.618124 10863 ts_tablet_manager.cc:1367] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: Time spent bootstrapping tablet: real 0.303s user 0.076s sys 0.001s
I20240501 19:56:25.618644 10863 raft_consensus.cc:348] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:25.619454 10863 raft_consensus.cc:729] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9564f3b39b549c1bd995354fa135c5f, State: Initialized, Role: LEARNER
I20240501 19:56:25.619585 10863 consensus_queue.cc:260] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 69, Last appended: 1.70, Last appended by leader: 70, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 49 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:25.620662 10642 raft_consensus.cc:1050] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: attempting to promote NON_VOTER 14798909ff724444a8be03c45c41929a to VOTER
I20240501 19:56:25.621066 10642 consensus_queue.cc:237] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.85, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } }
I20240501 19:56:25.622412 10863 ts_tablet_manager.cc:1398] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:25.622968 8966 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240501 19:56:25.624548 8859 tablet_copy_service.cc:342] P 0f2b49c6d3dc49fabee40802e73ac982: Request end of tablet copy session c9564f3b39b549c1bd995354fa135c5f-df35daed2f72400086ccd0b8c47ecab1 received from {username='slave'} at 127.5.139.68:56291
I20240501 19:56:25.624646 8859 tablet_copy_service.cc:433] P 0f2b49c6d3dc49fabee40802e73ac982: ending tablet copy session c9564f3b39b549c1bd995354fa135c5f-df35daed2f72400086ccd0b8c47ecab1 on tablet df35daed2f72400086ccd0b8c47ecab1 with peer c9564f3b39b549c1bd995354fa135c5f
I20240501 19:56:25.624704 9902 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
I20240501 19:56:25.625661 10510 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 86. (index mismatch)
I20240501 19:56:25.626870 9902 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 86, Last known committed idx: 84, Time since last communication: 0.000s
W20240501 19:56:25.628124 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.629904 10646 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.86: config changed from index 59 to 86, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:25.630518 8967 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index 59 to 86, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:25.633186 8691 catalog_manager.cc:5484] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 59 to 86, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.633319 10510 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Committing config change with OpId 1.86: config changed from index 59 to 86, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 86 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:25.639518 9100 consensus_queue.cc:237] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 86, Committed index: 86, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } }
I20240501 19:56:25.640993 10510 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 86. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240501 19:56:25.641924 9902 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 86, Time since last communication: 0.000s
I20240501 19:56:25.644331 10642 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index 86 to 87, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:25.644822 10845 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap replayed 1/1 log segments. Stats: ops{read=64 overwritten=0 applied=64 ignored=0} inserts{seen=695 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:25.645119 10845 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap complete.
I20240501 19:56:25.645311 10845 ts_tablet_manager.cc:1367] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent bootstrapping tablet: real 0.288s user 0.064s sys 0.008s
I20240501 19:56:25.645419 10649 raft_consensus.cc:1050] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982: attempting to promote NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b to VOTER
I20240501 19:56:25.645740 10649 consensus_queue.cc:237] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 81, Committed index: 81, Last appended: 1.81, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:25.645812 10845 raft_consensus.cc:348] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:25.645920 10845 raft_consensus.cc:729] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6977f9a52e9b4d06bcb12f9bee61553b, State: Initialized, Role: LEARNER
I20240501 19:56:25.646040 10845 consensus_queue.cc:260] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 64, Last appended: 1.64, Last appended by leader: 64, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 53 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:25.646296 10845 ts_tablet_manager.cc:1398] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:25.646692 10510 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index 86 to 87, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:25.648221 8684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49f4147edb7349bcbf51994e59501d19 with cas_config_opid_index 86: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.648557 9121 tablet_copy_service.cc:342] P b3b45c9058d84b4e9ace42c9154ab0e0: Request end of tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-aa477b2bd34d4b74b245314148e4851d received from {username='slave'} at 127.5.139.69:33027
I20240501 19:56:25.648662 9121 tablet_copy_service.cc:433] P b3b45c9058d84b4e9ace42c9154ab0e0: ending tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-aa477b2bd34d4b74b245314148e4851d on tablet aa477b2bd34d4b74b245314148e4851d with peer 6977f9a52e9b4d06bcb12f9bee61553b
I20240501 19:56:25.649499 8696 catalog_manager.cc:5484] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 86 to 87, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.650943 8969 raft_consensus.cc:1261] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
W20240501 19:56:25.651609 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.652781 8949 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f4147edb7349bcbf51994e59501d19 with delete_type TABLET_DATA_TOMBSTONED (TS ac13bdb319c448dba072fb9e62d6893b not found in new config with opid_index 87) from {username='slave'} at 127.0.0.1:45626
I20240501 19:56:25.652971 10964 tablet_replica.cc:330] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b: stopping tablet replica
I20240501 19:56:25.653141 10964 raft_consensus.cc:2229] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.653270 10964 raft_consensus.cc:2258] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.653965 10653 consensus_queue.cc:1035] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 19:56:25.654251 10964 ts_tablet_manager.cc:1875] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.656581 10964 ts_tablet_manager.cc:1888] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.86
I20240501 19:56:25.656661 10964 log.cc:1196] T 49f4147edb7349bcbf51994e59501d19 P ac13bdb319c448dba072fb9e62d6893b: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/49f4147edb7349bcbf51994e59501d19
I20240501 19:56:25.657490 10251 raft_consensus.cc:1261] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 81. Preceding OpId from leader: term: 1 index: 82. (index mismatch)
I20240501 19:56:25.657866 8682 catalog_manager.cc:4830] TS ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725): tablet 49f4147edb7349bcbf51994e59501d19 (table rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]) successfully deleted
I20240501 19:56:25.659106 9923 consensus_queue.cc:1035] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 82, Last known committed idx: 81, Time since last communication: 0.000s
I20240501 19:56:25.659731 9923 raft_consensus.cc:2932] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.82: config changed from index 49 to 82, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.661139 10251 raft_consensus.cc:2932] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 49 to 82, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.663241 8967 raft_consensus.cc:2932] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Committing config change with OpId 1.82: config changed from index 49 to 82, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.663870 8693 catalog_manager.cc:5484] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b reported cstate change: config changed from index 49 to 82, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 82 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.675594 8839 consensus_queue.cc:237] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
W20240501 19:56:25.676565 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.679355 10251 raft_consensus.cc:1261] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240501 19:56:25.683691 10738 consensus_queue.cc:1035] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240501 19:56:25.687075 10649 raft_consensus.cc:2932] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.689903 8681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0c4e313b03cc44d0a2f7864f7e9d4f90 with cas_config_opid_index 82: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.690469 10251 raft_consensus.cc:2932] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 82 to 84, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.692416 8691 catalog_manager.cc:5484] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index 82 to 84, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:25.696257 8949 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c4e313b03cc44d0a2f7864f7e9d4f90 with delete_type TABLET_DATA_TOMBSTONED (TS ac13bdb319c448dba072fb9e62d6893b not found in new config with opid_index 84) from {username='slave'} at 127.0.0.1:45626
I20240501 19:56:25.696563 10964 tablet_replica.cc:330] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: stopping tablet replica
I20240501 19:56:25.696692 10964 raft_consensus.cc:2229] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.696805 10964 raft_consensus.cc:2258] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.698112 10964 ts_tablet_manager.cc:1875] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.700141 10964 ts_tablet_manager.cc:1888] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.83
I20240501 19:56:25.700215 10964 log.cc:1196] T 0c4e313b03cc44d0a2f7864f7e9d4f90 P ac13bdb319c448dba072fb9e62d6893b: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/0c4e313b03cc44d0a2f7864f7e9d4f90
I20240501 19:56:25.700722 8682 catalog_manager.cc:4830] TS ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725): tablet 0c4e313b03cc44d0a2f7864f7e9d4f90 (table rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]) successfully deleted
W20240501 19:56:25.700917 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
W20240501 19:56:25.727129 9061 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.738760 10642 raft_consensus.cc:1050] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0: attempting to promote NON_VOTER c9564f3b39b549c1bd995354fa135c5f to VOTER
I20240501 19:56:25.739136 10642 consensus_queue.cc:237] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.63, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:25.740329 10121 raft_consensus.cc:1261] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240501 19:56:25.740810 8837 raft_consensus.cc:1261] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 63. Preceding OpId from leader: term: 1 index: 64. (index mismatch)
I20240501 19:56:25.746605 9902 consensus_queue.cc:1035] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 19:56:25.748503 10630 consensus_queue.cc:1035] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 64, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 19:56:25.753638 10630 raft_consensus.cc:2932] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.64: config changed from index 28 to 64, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.755092 10382 raft_consensus.cc:1203] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.45->[1.46-1.77] Dedup: 1.64->[1.65-1.77]
W20240501 19:56:25.757264 9060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Service unavailable: leader transfer in progress
I20240501 19:56:25.759068 10121 raft_consensus.cc:2932] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 28 to 64, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.761142 8691 catalog_manager.cc:5484] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 28 to 64, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.764516 8837 raft_consensus.cc:2932] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Committing config change with OpId 1.64: config changed from index 28 to 64, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 64 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.764688 9100 consensus_queue.cc:237] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 64, Committed index: 64, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:25.767745 10121 raft_consensus.cc:1261] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240501 19:56:25.768564 10630 raft_consensus.cc:979] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: : Instructing follower 6b4f42453a864314adc86aba2c3b3556 to start an election
I20240501 19:56:25.768690 10642 raft_consensus.cc:1067] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Signalling peer 6b4f42453a864314adc86aba2c3b3556 to start an election
I20240501 19:56:25.768992 10642 consensus_queue.cc:1035] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 64, Time since last communication: 0.000s
I20240501 19:56:25.769258 10382 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58979b7e47734549bc48710ff1d2b1c1"
dest_uuid: "6b4f42453a864314adc86aba2c3b3556"
from {username='slave'} at 127.5.139.67:49337
I20240501 19:56:25.769447 10382 raft_consensus.cc:482] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:25.769567 10382 raft_consensus.cc:3037] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:25.770594 10382 raft_consensus.cc:504] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:25.772091 10382 leader_election.cc:288] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [CANDIDATE]: Term 2 election: Requested vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731), b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:25.773273 9100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58979b7e47734549bc48710ff1d2b1c1" candidate_uuid: "6b4f42453a864314adc86aba2c3b3556" candidate_term: 2 candidate_status { last_received { term: 1 index: 65 } } ignore_live_leader: true dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
I20240501 19:56:25.773386 9100 raft_consensus.cc:3032] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:25.773437 9100 raft_consensus.cc:729] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:25.773576 9100 consensus_queue.cc:260] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:25.773725 9100 raft_consensus.cc:3037] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:25.774595 9100 raft_consensus.cc:2454] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4f42453a864314adc86aba2c3b3556 in term 2.
I20240501 19:56:25.776461 10319 leader_election.cc:302] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b4f42453a864314adc86aba2c3b3556, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:25.776585 8837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58979b7e47734549bc48710ff1d2b1c1" candidate_uuid: "6b4f42453a864314adc86aba2c3b3556" candidate_term: 2 candidate_status { last_received { term: 1 index: 65 } } ignore_live_leader: true dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
I20240501 19:56:25.776684 8837 raft_consensus.cc:3037] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:25.776703 10869 raft_consensus.cc:2781] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:25.776855 10869 raft_consensus.cc:686] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 2 LEADER]: Becoming Leader. State: Replica: 6b4f42453a864314adc86aba2c3b3556, State: Running, Role: LEADER
I20240501 19:56:25.777153 10869 consensus_queue.cc:237] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 65, Committed index: 65, Last appended: 1.65, Last appended by leader: 65, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:25.777673 8837 raft_consensus.cc:2454] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4f42453a864314adc86aba2c3b3556 in term 2.
I20240501 19:56:25.779269 8696 catalog_manager.cc:5484] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 reported cstate change: term changed from 1 to 2, leader changed from b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) to 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70). New cstate: current_term: 2 leader_uuid: "6b4f42453a864314adc86aba2c3b3556" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.781891 10802 raft_consensus.cc:2932] T 74706b357b2b43549757d24fa9a9d282 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index 64 to 65, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.783053 10121 raft_consensus.cc:2932] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 64 to 65, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.783466 8684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 74706b357b2b43549757d24fa9a9d282 with cas_config_opid_index 64: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.785478 8693 catalog_manager.cc:5484] T 74706b357b2b43549757d24fa9a9d282 P c9564f3b39b549c1bd995354fa135c5f reported cstate change: config changed from index 64 to 65, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.788153 8819 tablet_service.cc:1514] Processing DeleteTablet for tablet 74706b357b2b43549757d24fa9a9d282 with delete_type TABLET_DATA_TOMBSTONED (TS 0f2b49c6d3dc49fabee40802e73ac982 not found in new config with opid_index 65) from {username='slave'} at 127.0.0.1:48934
I20240501 19:56:25.788341 10966 tablet_replica.cc:330] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982: stopping tablet replica
I20240501 19:56:25.788471 10966 raft_consensus.cc:2229] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.788795 10966 raft_consensus.cc:2258] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.789789 10966 ts_tablet_manager.cc:1875] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.791893 10966 ts_tablet_manager.cc:1888] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.64
I20240501 19:56:25.791965 10966 log.cc:1196] T 74706b357b2b43549757d24fa9a9d282 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/74706b357b2b43549757d24fa9a9d282
I20240501 19:56:25.792372 8681 catalog_manager.cc:4830] TS 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731): tablet 74706b357b2b43549757d24fa9a9d282 (table rebalance_test_table_3 [id=8a5200c5824e48fd9493fa155a0a6e47]) successfully deleted
W20240501 19:56:25.793370 9060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45010: Illegal state: replica b3b45c9058d84b4e9ace42c9154ab0e0 is not leader of this config: current role FOLLOWER
I20240501 19:56:25.794440 10251 raft_consensus.cc:1203] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.52->[1.53-1.81] Dedup: 1.64->[1.65-1.81]
W20240501 19:56:25.825171 8798 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42232: Illegal state: replica 0f2b49c6d3dc49fabee40802e73ac982 is not leader of this config: current role FOLLOWER
I20240501 19:56:25.828917 10802 raft_consensus.cc:1050] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: attempting to promote NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b to VOTER
I20240501 19:56:25.829335 10802 consensus_queue.cc:237] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.82, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:25.831199 8967 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240501 19:56:25.831635 10802 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240501 19:56:25.832202 10252 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 82. Preceding OpId from leader: term: 1 index: 83. (index mismatch)
I20240501 19:56:25.832577 10642 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 83, Last known committed idx: 82, Time since last communication: 0.000s
I20240501 19:56:25.833614 10642 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.83: config changed from index 53 to 83, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.834476 8967 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 53 to 83, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.838696 10590 raft_consensus.cc:1050] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: attempting to promote NON_VOTER 6b4f42453a864314adc86aba2c3b3556 to VOTER
I20240501 19:56:25.838879 8696 catalog_manager.cc:5484] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index 53 to 83, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.839026 10590 consensus_queue.cc:237] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:25.841068 10383 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 19:56:25.841428 10615 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 77, Time since last communication: 0.000s
I20240501 19:56:25.842135 9100 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 19:56:25.842559 10616 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 19:56:25.844043 10615 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index 46 to 79, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.844633 10382 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 46 to 79, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.846104 9101 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index 46 to 79, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.846900 8691 catalog_manager.cc:5484] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 reported cstate change: config changed from index 46 to 79, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.853953 8969 consensus_queue.cc:237] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 80, Committed index: 80, Last appended: 1.80, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:25.854916 10382 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 80. Preceding OpId from leader: term: 1 index: 81. (index mismatch)
I20240501 19:56:25.855281 10590 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 81, Last known committed idx: 80, Time since last communication: 0.000s
I20240501 19:56:25.857367 10590 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.81: config changed from index 79 to 81, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.859246 10382 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Committing config change with OpId 1.81: config changed from index 79 to 81, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.860625 8682 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68c8311312bd4b50bdbedf414ada6bd2 with cas_config_opid_index 79: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.862128 8691 catalog_manager.cc:5484] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index 79 to 81, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 81 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.862485 10252 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.83: config changed from index 53 to 83, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 83 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.864585 9080 tablet_service.cc:1514] Processing DeleteTablet for tablet 68c8311312bd4b50bdbedf414ada6bd2 with delete_type TABLET_DATA_TOMBSTONED (TS b3b45c9058d84b4e9ace42c9154ab0e0 not found in new config with opid_index 81) from {username='slave'} at 127.0.0.1:44930
I20240501 19:56:25.864778 10967 tablet_replica.cc:330] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: stopping tablet replica
I20240501 19:56:25.864892 10967 raft_consensus.cc:2229] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.864988 10967 raft_consensus.cc:2258] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.865006 9100 consensus_queue.cc:237] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:25.865778 10967 ts_tablet_manager.cc:1875] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.867388 10121 raft_consensus.cc:1203] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.48->[1.49-1.82] Dedup: 1.70->[1.71-1.82]
I20240501 19:56:25.867859 10967 ts_tablet_manager.cc:1888] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.80
I20240501 19:56:25.867934 10967 log.cc:1196] T 68c8311312bd4b50bdbedf414ada6bd2 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/68c8311312bd4b50bdbedf414ada6bd2
I20240501 19:56:25.868534 8684 catalog_manager.cc:4830] TS b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389): tablet 68c8311312bd4b50bdbedf414ada6bd2 (table rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]) successfully deleted
I20240501 19:56:25.876731 10252 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240501 19:56:25.878783 10630 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20240501 19:56:25.882061 10734 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index 83 to 85, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.883498 8684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa477b2bd34d4b74b245314148e4851d with cas_config_opid_index 83: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.884721 8694 catalog_manager.cc:5484] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 83 to 85, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.886174 10252 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 83 to 85, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:25.887136 8949 tablet_service.cc:1514] Processing DeleteTablet for tablet aa477b2bd34d4b74b245314148e4851d with delete_type TABLET_DATA_TOMBSTONED (TS ac13bdb319c448dba072fb9e62d6893b not found in new config with opid_index 85) from {username='slave'} at 127.0.0.1:45626
I20240501 19:56:25.887328 10964 tablet_replica.cc:330] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b: stopping tablet replica
I20240501 19:56:25.887441 10964 raft_consensus.cc:2229] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.887775 10964 raft_consensus.cc:2258] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.888551 10964 ts_tablet_manager.cc:1875] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.890640 10964 ts_tablet_manager.cc:1888] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.84
I20240501 19:56:25.890687 10964 log.cc:1196] T aa477b2bd34d4b74b245314148e4851d P ac13bdb319c448dba072fb9e62d6893b: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa477b2bd34d4b74b245314148e4851d
I20240501 19:56:25.891464 8682 catalog_manager.cc:4830] TS ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725): tablet aa477b2bd34d4b74b245314148e4851d (table rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]) successfully deleted
I20240501 19:56:25.911342 8837 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Refusing update from remote peer 6b4f42453a864314adc86aba2c3b3556: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 2 index: 67. (index mismatch)
I20240501 19:56:25.911914 9100 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Refusing update from remote peer 6b4f42453a864314adc86aba2c3b3556: Log matching property violated. Preceding OpId in replica: term: 1 index: 65. Preceding OpId from leader: term: 2 index: 67. (index mismatch)
I20240501 19:56:25.911865 10869 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240501 19:56:25.912752 10738 raft_consensus.cc:1050] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: attempting to promote NON_VOTER c9564f3b39b549c1bd995354fa135c5f to VOTER
I20240501 19:56:25.913123 10738 consensus_queue.cc:237] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 83, Committed index: 83, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:25.913118 10869 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 66, Last known committed idx: 65, Time since last communication: 0.000s
I20240501 19:56:25.914216 9101 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240501 19:56:25.914898 10120 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240501 19:56:25.915176 10649 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 83, Time since last communication: 0.000s
I20240501 19:56:25.916813 10902 mvcc.cc:205] Tried to move back new op lower bound from 7022974508688064512 to 7022974508146159616. Current Snapshot: MvccSnapshot[applied={T|T < 7022974508688064512}]
I20240501 19:56:25.921499 9923 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 82, Time since last communication: 0.000s
I20240501 19:56:25.923357 10738 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index 49 to 84, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.925693 9101 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 49 to 84, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.931412 10121 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index 49 to 84, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.933373 8694 catalog_manager.cc:5484] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 49 to 84, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.940230 8839 consensus_queue.cc:237] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 84, Committed index: 84, Last appended: 1.84, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:25.941191 10121 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 84. Preceding OpId from leader: term: 1 index: 85. (index mismatch)
I20240501 19:56:25.941934 10653 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 85, Last known committed idx: 84, Time since last communication: 0.000s
I20240501 19:56:25.944171 10653 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.85: config changed from index 84 to 85, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.946574 8681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df35daed2f72400086ccd0b8c47ecab1 with cas_config_opid_index 84: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.946894 10121 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Committing config change with OpId 1.85: config changed from index 84 to 85, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:25.948465 8696 catalog_manager.cc:5484] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index 84 to 85, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 85 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.952162 9080 tablet_service.cc:1514] Processing DeleteTablet for tablet df35daed2f72400086ccd0b8c47ecab1 with delete_type TABLET_DATA_TOMBSTONED (TS b3b45c9058d84b4e9ace42c9154ab0e0 not found in new config with opid_index 85) from {username='slave'} at 127.0.0.1:44930
I20240501 19:56:25.952344 10967 tablet_replica.cc:330] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0: stopping tablet replica
I20240501 19:56:25.952471 10967 raft_consensus.cc:2229] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.952594 10967 raft_consensus.cc:2258] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.953402 10967 ts_tablet_manager.cc:1875] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.955356 10967 ts_tablet_manager.cc:1888] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.84
I20240501 19:56:25.955420 10967 log.cc:1196] T df35daed2f72400086ccd0b8c47ecab1 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/df35daed2f72400086ccd0b8c47ecab1
I20240501 19:56:25.955852 8684 catalog_manager.cc:4830] TS b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389): tablet df35daed2f72400086ccd0b8c47ecab1 (table rebalance_test_table_1 [id=3c90b69a201147fa8697594a2940e3a2]) successfully deleted
I20240501 19:56:25.962363 10382 consensus_queue.cc:237] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 2.69, Last appended by leader: 65, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:25.964838 8837 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Refusing update from remote peer 6b4f42453a864314adc86aba2c3b3556: Log matching property violated. Preceding OpId in replica: term: 2 index: 69. Preceding OpId from leader: term: 2 index: 70. (index mismatch)
I20240501 19:56:25.965196 10996 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240501 19:56:25.969020 10996 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 2 LEADER]: Committing config change with OpId 2.70: config changed from index 63 to 70, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.970839 8681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 58979b7e47734549bc48710ff1d2b1c1 with cas_config_opid_index 63: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:25.971567 8837 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Committing config change with OpId 2.70: config changed from index 63 to 70, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:25.971752 8696 catalog_manager.cc:5484] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 reported cstate change: config changed from index 63 to 70, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New cstate: current_term: 2 leader_uuid: "6b4f42453a864314adc86aba2c3b3556" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:25.975492 9080 tablet_service.cc:1514] Processing DeleteTablet for tablet 58979b7e47734549bc48710ff1d2b1c1 with delete_type TABLET_DATA_TOMBSTONED (TS b3b45c9058d84b4e9ace42c9154ab0e0 not found in new config with opid_index 70) from {username='slave'} at 127.0.0.1:44930
I20240501 19:56:25.975714 10967 tablet_replica.cc:330] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: stopping tablet replica
I20240501 19:56:25.975841 10967 raft_consensus.cc:2229] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:25.975931 10967 pending_rounds.cc:62] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: Trying to abort 1 pending ops.
I20240501 19:56:25.976052 10967 pending_rounds.cc:68] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: Aborting op as it isn't in flight: id { term: 2 index: 68 } timestamp: 7022974508769210368 op_type: WRITE_OP write_request { tablet_id: "58979b7e47734549bc48710ff1d2b1c1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\031\2356\016\006,\306\245\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""13c265eeff642e6f" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022974508724879360 authz_token { token_data: "\010\223\277\312\261\006\"3\n\005slave\022*\n 8a5200c5824e48fd9493fa155a0a6e47\020\001\030\001 \001(\001" signature: "<redacted>""2dE\032Z:\006\253\211\252\203\206e\nG\221\276^nqSi\316R\303x\034<\324\255\n\274wz\000\024\273\017\214k\256\014\331\022\\\355%9\033\022\235\354\006\032M2\215o\253s\213\'d*" signing_key_seq_num: 0 } } request_id { client_id: "1d92d292d5294d869ddcb4aff640f29a" seq_no: 183 first_incomplete_seq_no: 183 attempt_no: 0 }
I20240501 19:56:25.976791 10967 raft_consensus.cc:2258] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:25.977919 10967 ts_tablet_manager.cc:1875] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:25.991906 10967 ts_tablet_manager.cc:1888] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.68
I20240501 19:56:25.992048 10967 log.cc:1196] T 58979b7e47734549bc48710ff1d2b1c1 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/58979b7e47734549bc48710ff1d2b1c1
I20240501 19:56:25.992941 8684 catalog_manager.cc:4830] TS b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389): tablet 58979b7e47734549bc48710ff1d2b1c1 (table rebalance_test_table_3 [id=8a5200c5824e48fd9493fa155a0a6e47]) successfully deleted
I20240501 19:56:26.335593 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:26.337201 10495 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:26.348292 8949 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:26.354984 10232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:26.382326 8819 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:26.386451 9080 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:26.410535 10101 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:26.570961 9101 consensus_queue.cc:237] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 122, Committed index: 122, Last appended: 1.122, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:26.572314 10510 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 122. Preceding OpId from leader: term: 1 index: 123. (index mismatch)
I20240501 19:56:26.573421 9902 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 123, Last known committed idx: 121, Time since last communication: 0.000s
W20240501 19:56:26.574668 9034 consensus_peers.cc:485] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 -> Peer c9564f3b39b549c1bd995354fa135c5f (127.5.139.68:46413): Couldn't send request to peer c9564f3b39b549c1bd995354fa135c5f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49f4147edb7349bcbf51994e59501d19. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:26.576782 10734 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.123: config changed from index 87 to 123, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
I20240501 19:56:26.579061 10510 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Committing config change with OpId 1.123: config changed from index 87 to 123, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New config: { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } } }
I20240501 19:56:26.581143 8691 catalog_manager.cc:5484] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 87 to 123, NON_VOTER c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) added. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:26.634019 8969 consensus_queue.cc:237] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 1.105, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } }
W20240501 19:56:26.635532 8904 consensus_peers.cc:485] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b -> Peer 14798909ff724444a8be03c45c41929a (127.5.139.71:38393): Couldn't send request to peer 14798909ff724444a8be03c45c41929a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 68c8311312bd4b50bdbedf414ada6bd2. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:26.635862 10383 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 105. Preceding OpId from leader: term: 1 index: 106. (index mismatch)
I20240501 19:56:26.637281 9986 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240501 19:56:26.640869 11008 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.106: config changed from index 81 to 106, NON_VOTER 14798909ff724444a8be03c45c41929a (127.5.139.71) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } } }
I20240501 19:56:26.641186 10383 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Committing config change with OpId 1.106: config changed from index 81 to 106, NON_VOTER 14798909ff724444a8be03c45c41929a (127.5.139.71) added. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } } }
I20240501 19:56:26.643998 8693 catalog_manager.cc:5484] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index 81 to 106, NON_VOTER 14798909ff724444a8be03c45c41929a (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:26.677835 10383 consensus_queue.cc:237] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 2.98, Last appended by leader: 65, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:26.679265 8837 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Refusing update from remote peer 6b4f42453a864314adc86aba2c3b3556: Log matching property violated. Preceding OpId in replica: term: 2 index: 98. Preceding OpId from leader: term: 2 index: 99. (index mismatch)
I20240501 19:56:26.679630 11101 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 19:56:26.681020 10869 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 2 LEADER]: Committing config change with OpId 2.99: config changed from index 70 to 99, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:26.681488 8837 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Committing config change with OpId 2.99: config changed from index 70 to 99, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:26.684257 8691 catalog_manager.cc:5484] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 reported cstate change: config changed from index 70 to 99, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New cstate: current_term: 2 leader_uuid: "6b4f42453a864314adc86aba2c3b3556" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:26.691901 10316 consensus_peers.cc:485] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 -> Peer 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483): Couldn't send request to peer 6977f9a52e9b4d06bcb12f9bee61553b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58979b7e47734549bc48710ff1d2b1c1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:26.751063 8839 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 1.117, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:26.755072 10251 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 19:56:26.756256 10649 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 116, Time since last communication: 0.000s
I20240501 19:56:26.759673 10654 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.118: config changed from index 74 to 118, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } } }
I20240501 19:56:26.760129 10252 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 74 to 118, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } } }
I20240501 19:56:26.762538 8693 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index 74 to 118, NON_VOTER 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:26.776271 8772 consensus_peers.cc:485] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 -> Peer 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441): Couldn't send request to peer 6b4f42453a864314adc86aba2c3b3556. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d827d0b0d5ba4969908768c96f92c1f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:26.783568 8837 consensus_queue.cc:237] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:26.785387 10120 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240501 19:56:26.786161 10649 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
W20240501 19:56:26.787973 8772 consensus_peers.cc:485] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 -> Peer 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483): Couldn't send request to peer 6977f9a52e9b4d06bcb12f9bee61553b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df35daed2f72400086ccd0b8c47ecab1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:26.788635 10738 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.111: config changed from index 85 to 111, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:26.789109 10120 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 85 to 111, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:26.792533 8693 catalog_manager.cc:5484] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f reported cstate change: config changed from index 85 to 111, NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } } }
I20240501 19:56:27.163684 11181 ts_tablet_manager.cc:902] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: Initiating tablet copy from peer 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:27.164701 11181 tablet_copy_client.cc:308] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38731
I20240501 19:56:27.165416 8859 tablet_copy_service.cc:140] P 0f2b49c6d3dc49fabee40802e73ac982: Received BeginTabletCopySession request for tablet d827d0b0d5ba4969908768c96f92c1f3 from peer 6b4f42453a864314adc86aba2c3b3556 ({username='slave'} at 127.5.139.70:34955)
I20240501 19:56:27.165539 8859 tablet_copy_service.cc:161] P 0f2b49c6d3dc49fabee40802e73ac982: Beginning new tablet copy session on tablet d827d0b0d5ba4969908768c96f92c1f3 from peer 6b4f42453a864314adc86aba2c3b3556 at {username='slave'} at 127.5.139.70:34955: session id = 6b4f42453a864314adc86aba2c3b3556-d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:27.166589 8859 tablet_copy_source_session.cc:205] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:27.168798 11181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d827d0b0d5ba4969908768c96f92c1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.175839 11186 ts_tablet_manager.cc:902] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: Initiating tablet copy from peer ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:27.177091 11181 tablet_copy_client.cc:783] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:27.177381 11181 tablet_copy_client.cc:647] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:27.177922 11186 tablet_copy_client.cc:308] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:43725
I20240501 19:56:27.178393 8990 tablet_copy_service.cc:140] P ac13bdb319c448dba072fb9e62d6893b: Received BeginTabletCopySession request for tablet 68c8311312bd4b50bdbedf414ada6bd2 from peer 14798909ff724444a8be03c45c41929a ({username='slave'} at 127.5.139.71:48051)
I20240501 19:56:27.178484 8990 tablet_copy_service.cc:161] P ac13bdb319c448dba072fb9e62d6893b: Beginning new tablet copy session on tablet 68c8311312bd4b50bdbedf414ada6bd2 from peer 14798909ff724444a8be03c45c41929a at {username='slave'} at 127.5.139.71:48051: session id = 14798909ff724444a8be03c45c41929a-68c8311312bd4b50bdbedf414ada6bd2
I20240501 19:56:27.179406 8990 tablet_copy_source_session.cc:205] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:27.186607 11186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68c8311312bd4b50bdbedf414ada6bd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.186592 11195 ts_tablet_manager.cc:902] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: Initiating tablet copy from peer b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:27.187321 11195 tablet_copy_client.cc:308] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:35389
I20240501 19:56:27.187729 9121 tablet_copy_service.cc:140] P b3b45c9058d84b4e9ace42c9154ab0e0: Received BeginTabletCopySession request for tablet 49f4147edb7349bcbf51994e59501d19 from peer c9564f3b39b549c1bd995354fa135c5f ({username='slave'} at 127.5.139.68:59875)
I20240501 19:56:27.187827 9121 tablet_copy_service.cc:161] P b3b45c9058d84b4e9ace42c9154ab0e0: Beginning new tablet copy session on tablet 49f4147edb7349bcbf51994e59501d19 from peer c9564f3b39b549c1bd995354fa135c5f at {username='slave'} at 127.5.139.68:59875: session id = c9564f3b39b549c1bd995354fa135c5f-49f4147edb7349bcbf51994e59501d19
I20240501 19:56:27.188618 9121 tablet_copy_source_session.cc:205] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:27.194849 11195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49f4147edb7349bcbf51994e59501d19. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.196744 11181 tablet_copy_client.cc:523] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:27.198758 11181 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: Bootstrap starting.
I20240501 19:56:27.204712 11186 tablet_copy_client.cc:783] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:27.205730 11186 tablet_copy_client.cc:647] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:27.207420 11186 tablet_copy_client.cc:523] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:27.213217 11195 tablet_copy_client.cc:783] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:27.213371 11195 tablet_copy_client.cc:647] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:27.214145 11202 ts_tablet_manager.cc:902] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: Initiating tablet copy from peer 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441)
I20240501 19:56:27.214672 11202 tablet_copy_client.cc:308] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.70:43441
I20240501 19:56:27.215458 11195 tablet_copy_client.cc:523] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:27.220546 11186 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: Bootstrap starting.
I20240501 19:56:27.238173 10403 tablet_copy_service.cc:140] P 6b4f42453a864314adc86aba2c3b3556: Received BeginTabletCopySession request for tablet 58979b7e47734549bc48710ff1d2b1c1 from peer 6977f9a52e9b4d06bcb12f9bee61553b ({username='slave'} at 127.5.139.69:59113)
I20240501 19:56:27.238323 10403 tablet_copy_service.cc:161] P 6b4f42453a864314adc86aba2c3b3556: Beginning new tablet copy session on tablet 58979b7e47734549bc48710ff1d2b1c1 from peer 6977f9a52e9b4d06bcb12f9bee61553b at {username='slave'} at 127.5.139.69:59113: session id = 6977f9a52e9b4d06bcb12f9bee61553b-58979b7e47734549bc48710ff1d2b1c1
I20240501 19:56:27.239753 10403 tablet_copy_source_session.cc:205] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:27.250193 11202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58979b7e47734549bc48710ff1d2b1c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.251787 11195 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap starting.
I20240501 19:56:27.272478 11202 tablet_copy_client.cc:783] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:27.277029 11202 tablet_copy_client.cc:647] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:27.279048 11202 tablet_copy_client.cc:523] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:27.280483 11202 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap starting.
I20240501 19:56:27.332623 11218 ts_tablet_manager.cc:902] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: Initiating tablet copy from peer 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731)
I20240501 19:56:27.333376 11218 tablet_copy_client.cc:308] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38731
I20240501 19:56:27.333958 8859 tablet_copy_service.cc:140] P 0f2b49c6d3dc49fabee40802e73ac982: Received BeginTabletCopySession request for tablet df35daed2f72400086ccd0b8c47ecab1 from peer 6977f9a52e9b4d06bcb12f9bee61553b ({username='slave'} at 127.5.139.69:52827)
I20240501 19:56:27.334074 8859 tablet_copy_service.cc:161] P 0f2b49c6d3dc49fabee40802e73ac982: Beginning new tablet copy session on tablet df35daed2f72400086ccd0b8c47ecab1 from peer 6977f9a52e9b4d06bcb12f9bee61553b at {username='slave'} at 127.5.139.69:52827: session id = 6977f9a52e9b4d06bcb12f9bee61553b-df35daed2f72400086ccd0b8c47ecab1
I20240501 19:56:27.334923 8859 tablet_copy_source_session.cc:205] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:27.335745 11218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df35daed2f72400086ccd0b8c47ecab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:27.340072 11218 tablet_copy_client.cc:783] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:27.340219 11218 tablet_copy_client.cc:647] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:27.368525 11218 tablet_copy_client.cc:523] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:27.369987 11218 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap starting.
I20240501 19:56:27.476282 11202 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap replayed 1/1 log segments. Stats: ops{read=120 overwritten=0 applied=120 ignored=0} inserts{seen=747 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:27.476764 11202 tablet_bootstrap.cc:492] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap complete.
I20240501 19:56:27.476975 11202 ts_tablet_manager.cc:1367] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent bootstrapping tablet: real 0.197s user 0.099s sys 0.008s
I20240501 19:56:27.477485 11202 raft_consensus.cc:348] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:27.477619 11202 raft_consensus.cc:729] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6977f9a52e9b4d06bcb12f9bee61553b, State: Initialized, Role: LEARNER
I20240501 19:56:27.477720 11202 consensus_queue.cc:260] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 120, Last appended: 2.120, Last appended by leader: 120, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:27.478538 11202 ts_tablet_manager.cc:1398] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.479467 10403 tablet_copy_service.cc:342] P 6b4f42453a864314adc86aba2c3b3556: Request end of tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-58979b7e47734549bc48710ff1d2b1c1 received from {username='slave'} at 127.5.139.69:59113
I20240501 19:56:27.479568 10403 tablet_copy_service.cc:433] P 6b4f42453a864314adc86aba2c3b3556: ending tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-58979b7e47734549bc48710ff1d2b1c1 on tablet 58979b7e47734549bc48710ff1d2b1c1 with peer 6977f9a52e9b4d06bcb12f9bee61553b
I20240501 19:56:27.561698 11181 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: Bootstrap replayed 1/1 log segments. Stats: ops{read=129 overwritten=0 applied=128 ignored=0} inserts{seen=797 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:27.565913 11181 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: Bootstrap complete.
I20240501 19:56:27.566278 11181 ts_tablet_manager.cc:1367] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: Time spent bootstrapping tablet: real 0.368s user 0.109s sys 0.009s
I20240501 19:56:27.566874 11181 raft_consensus.cc:348] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:27.567344 11181 raft_consensus.cc:729] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b4f42453a864314adc86aba2c3b3556, State: Initialized, Role: LEARNER
I20240501 19:56:27.567517 11181 consensus_queue.cc:260] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 128, Last appended: 1.129, Last appended by leader: 129, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: true } }
I20240501 19:56:27.567884 11181 ts_tablet_manager.cc:1398] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.597687 11218 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap replayed 1/1 log segments. Stats: ops{read=133 overwritten=0 applied=133 ignored=0} inserts{seen=815 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:27.598161 11218 tablet_bootstrap.cc:492] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: Bootstrap complete.
I20240501 19:56:27.598551 11218 ts_tablet_manager.cc:1367] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent bootstrapping tablet: real 0.229s user 0.115s sys 0.006s
I20240501 19:56:27.599534 11218 raft_consensus.cc:348] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:27.599751 11218 raft_consensus.cc:729] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6977f9a52e9b4d06bcb12f9bee61553b, State: Initialized, Role: LEARNER
I20240501 19:56:27.599906 11218 consensus_queue.cc:260] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 133, Last appended: 1.133, Last appended by leader: 133, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: true } }
I20240501 19:56:27.600383 11218 ts_tablet_manager.cc:1398] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:27.601567 8859 tablet_copy_service.cc:342] P 0f2b49c6d3dc49fabee40802e73ac982: Request end of tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-df35daed2f72400086ccd0b8c47ecab1 received from {username='slave'} at 127.5.139.69:52827
I20240501 19:56:27.601661 8859 tablet_copy_service.cc:433] P 0f2b49c6d3dc49fabee40802e73ac982: ending tablet copy session 6977f9a52e9b4d06bcb12f9bee61553b-df35daed2f72400086ccd0b8c47ecab1 on tablet df35daed2f72400086ccd0b8c47ecab1 with peer 6977f9a52e9b4d06bcb12f9bee61553b
I20240501 19:56:27.604543 8859 tablet_copy_service.cc:342] P 0f2b49c6d3dc49fabee40802e73ac982: Request end of tablet copy session 6b4f42453a864314adc86aba2c3b3556-d827d0b0d5ba4969908768c96f92c1f3 received from {username='slave'} at 127.5.139.70:34955
I20240501 19:56:27.604619 8859 tablet_copy_service.cc:433] P 0f2b49c6d3dc49fabee40802e73ac982: ending tablet copy session 6b4f42453a864314adc86aba2c3b3556-d827d0b0d5ba4969908768c96f92c1f3 on tablet d827d0b0d5ba4969908768c96f92c1f3 with peer 6b4f42453a864314adc86aba2c3b3556
I20240501 19:56:27.649367 10252 raft_consensus.cc:1203] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.98->[2.99-2.132] Dedup: 2.120->[2.121-2.132]
I20240501 19:56:27.649832 10382 raft_consensus.cc:1203] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.117->[1.118-1.142] Dedup: 1.129->[1.130-1.142]
I20240501 19:56:27.655421 11186 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: Bootstrap replayed 1/1 log segments. Stats: ops{read=126 overwritten=0 applied=125 ignored=0} inserts{seen=753 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:27.655972 11186 tablet_bootstrap.cc:492] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: Bootstrap complete.
I20240501 19:56:27.656229 11186 ts_tablet_manager.cc:1367] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: Time spent bootstrapping tablet: real 0.436s user 0.118s sys 0.000s
I20240501 19:56:27.656770 11186 raft_consensus.cc:348] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } }
I20240501 19:56:27.657574 11186 raft_consensus.cc:729] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 14798909ff724444a8be03c45c41929a, State: Initialized, Role: LEARNER
I20240501 19:56:27.657757 11186 consensus_queue.cc:260] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 125, Last appended: 1.126, Last appended by leader: 126, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: true } }
I20240501 19:56:27.658114 11186 ts_tablet_manager.cc:1398] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:27.658571 8990 tablet_copy_service.cc:342] P ac13bdb319c448dba072fb9e62d6893b: Request end of tablet copy session 14798909ff724444a8be03c45c41929a-68c8311312bd4b50bdbedf414ada6bd2 received from {username='slave'} at 127.5.139.71:48051
I20240501 19:56:27.658655 8990 tablet_copy_service.cc:433] P ac13bdb319c448dba072fb9e62d6893b: ending tablet copy session 14798909ff724444a8be03c45c41929a-68c8311312bd4b50bdbedf414ada6bd2 on tablet 68c8311312bd4b50bdbedf414ada6bd2 with peer 14798909ff724444a8be03c45c41929a
I20240501 19:56:27.667745 11195 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=149 overwritten=0 applied=148 ignored=0} inserts{seen=830 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 19:56:27.668030 11195 tablet_bootstrap.cc:492] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: Bootstrap complete.
I20240501 19:56:27.668258 11195 ts_tablet_manager.cc:1367] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: Time spent bootstrapping tablet: real 0.424s user 0.116s sys 0.016s
I20240501 19:56:27.668723 11195 raft_consensus.cc:348] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:27.669090 11195 raft_consensus.cc:729] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c9564f3b39b549c1bd995354fa135c5f, State: Initialized, Role: LEARNER
I20240501 19:56:27.669443 11195 consensus_queue.cc:260] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 148, Last appended: 1.149, Last appended by leader: 149, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 123 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: true } }
I20240501 19:56:27.669816 11195 ts_tablet_manager.cc:1398] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:27.674952 9121 tablet_copy_service.cc:342] P b3b45c9058d84b4e9ace42c9154ab0e0: Request end of tablet copy session c9564f3b39b549c1bd995354fa135c5f-49f4147edb7349bcbf51994e59501d19 received from {username='slave'} at 127.5.139.68:59875
I20240501 19:56:27.675063 9121 tablet_copy_service.cc:433] P b3b45c9058d84b4e9ace42c9154ab0e0: ending tablet copy session c9564f3b39b549c1bd995354fa135c5f-49f4147edb7349bcbf51994e59501d19 on tablet 49f4147edb7349bcbf51994e59501d19 with peer c9564f3b39b549c1bd995354fa135c5f
I20240501 19:56:27.691735 10996 raft_consensus.cc:1050] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556: attempting to promote NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b to VOTER
I20240501 19:56:27.692365 10996 consensus_queue.cc:237] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 134, Committed index: 134, Last appended: 2.134, Last appended by leader: 65, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:27.694155 8837 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Refusing update from remote peer 6b4f42453a864314adc86aba2c3b3556: Log matching property violated. Preceding OpId in replica: term: 2 index: 134. Preceding OpId from leader: term: 2 index: 135. (index mismatch)
I20240501 19:56:27.694453 10251 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEARNER]: Refusing update from remote peer 6b4f42453a864314adc86aba2c3b3556: Log matching property violated. Preceding OpId in replica: term: 2 index: 133. Preceding OpId from leader: term: 2 index: 135. (index mismatch)
I20240501 19:56:27.694921 10869 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 134, Time since last communication: 0.000s
I20240501 19:56:27.701144 10869 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 135, Last known committed idx: 133, Time since last communication: 0.000s
I20240501 19:56:27.702072 11101 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 2 LEADER]: Committing config change with OpId 2.135: config changed from index 99 to 135, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:27.706857 10251 raft_consensus.cc:1203] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.133->[2.134-2.135] Dedup: 2.134->[2.135-2.135]
I20240501 19:56:27.707206 8691 catalog_manager.cc:5484] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 reported cstate change: config changed from index 99 to 135, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6b4f42453a864314adc86aba2c3b3556" committed_config { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:27.710987 8837 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Committing config change with OpId 2.135: config changed from index 99 to 135, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:27.724695 10251 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Committing config change with OpId 2.135: config changed from index 99 to 135, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 135 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:27.728713 10649 raft_consensus.cc:1050] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: attempting to promote NON_VOTER 6b4f42453a864314adc86aba2c3b3556 to VOTER
I20240501 19:56:27.729082 10649 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 145, Committed index: 145, Last appended: 1.145, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:27.731196 10251 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch)
I20240501 19:56:27.732930 10654 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s
I20240501 19:56:27.736939 10381 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 1 LEARNER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch)
I20240501 19:56:27.739259 10654 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s
I20240501 19:56:27.742343 10738 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.146: config changed from index 118 to 146, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:27.743885 10251 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 118 to 146, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:27.757530 8693 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index 118 to 146, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:27.758270 10381 consensus_queue.cc:237] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 137, Committed index: 137, Last appended: 2.138, Last appended by leader: 65, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:27.759488 10252 raft_consensus.cc:1261] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Refusing update from remote peer 6b4f42453a864314adc86aba2c3b3556: Log matching property violated. Preceding OpId in replica: term: 2 index: 138. Preceding OpId from leader: term: 2 index: 139. (index mismatch)
I20240501 19:56:27.760254 11101 consensus_queue.cc:1035] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 139, Last known committed idx: 137, Time since last communication: 0.000s
I20240501 19:56:27.765223 11101 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 [term 2 LEADER]: Committing config change with OpId 2.139: config changed from index 135 to 139, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:27.767673 8681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 58979b7e47734549bc48710ff1d2b1c1 with cas_config_opid_index 135: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:27.768064 8693 catalog_manager.cc:5484] T 58979b7e47734549bc48710ff1d2b1c1 P 6b4f42453a864314adc86aba2c3b3556 reported cstate change: config changed from index 135 to 139, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New cstate: current_term: 2 leader_uuid: "6b4f42453a864314adc86aba2c3b3556" committed_config { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:27.768188 10252 raft_consensus.cc:2932] T 58979b7e47734549bc48710ff1d2b1c1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Committing config change with OpId 2.139: config changed from index 135 to 139, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 139 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:27.778293 8819 tablet_service.cc:1514] Processing DeleteTablet for tablet 58979b7e47734549bc48710ff1d2b1c1 with delete_type TABLET_DATA_TOMBSTONED (TS 0f2b49c6d3dc49fabee40802e73ac982 not found in new config with opid_index 139) from {username='slave'} at 127.0.0.1:48934
I20240501 19:56:27.779654 10381 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 118 to 146, 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:27.800325 11264 tablet_replica.cc:330] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: stopping tablet replica
I20240501 19:56:27.802107 11264 raft_consensus.cc:2229] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:27.802242 11264 pending_rounds.cc:62] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: Trying to abort 1 pending ops.
I20240501 19:56:27.802325 11264 pending_rounds.cc:68] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: Aborting op as it isn't in flight: id { term: 2 index: 138 } timestamp: 7022974516232544256 op_type: WRITE_OP write_request { tablet_id: "58979b7e47734549bc48710ff1d2b1c1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\\\221\376w\243VF\366\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""39d49e2b6bb86d6c" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022974516206137344 authz_token { token_data: "\010\223\277\312\261\006\"3\n\005slave\022*\n 8a5200c5824e48fd9493fa155a0a6e47\020\001\030\001 \001(\001" signature: "<redacted>""2dE\032Z:\006\253\211\252\203\206e\nG\221\276^nqSi\316R\303x\034<\324\255\n\274wz\000\024\273\017\214k\256\014\331\022\\\355%9\033\022\235\354\006\032M2\215o\253s\213\'d*" signing_key_seq_num: 0 } } request_id { client_id: "1d92d292d5294d869ddcb4aff640f29a" seq_no: 374 first_incomplete_seq_no: 374 attempt_no: 0 }
I20240501 19:56:27.803586 11264 raft_consensus.cc:2258] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:27.804430 11264 ts_tablet_manager.cc:1875] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:27.807852 11264 ts_tablet_manager.cc:1888] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.138
I20240501 19:56:27.808013 11264 log.cc:1196] T 58979b7e47734549bc48710ff1d2b1c1 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/58979b7e47734549bc48710ff1d2b1c1
I20240501 19:56:27.810814 10119 raft_consensus.cc:1203] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.122->[1.123-1.169] Dedup: 1.149->[1.150-1.169]
I20240501 19:56:27.815001 8681 catalog_manager.cc:4830] TS 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731): tablet 58979b7e47734549bc48710ff1d2b1c1 (table rebalance_test_table_3 [id=8a5200c5824e48fd9493fa155a0a6e47]) successfully deleted
I20240501 19:56:27.869364 9902 raft_consensus.cc:1050] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: attempting to promote NON_VOTER c9564f3b39b549c1bd995354fa135c5f to VOTER
I20240501 19:56:27.869758 9902 consensus_queue.cc:237] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 170, Committed index: 170, Last appended: 1.170, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:27.870913 10121 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 1 LEARNER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240501 19:56:27.871284 10510 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 170. Preceding OpId from leader: term: 1 index: 171. (index mismatch)
I20240501 19:56:27.871650 10646 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s
I20240501 19:56:27.873482 10646 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.171: config changed from index 123 to 171, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:27.873772 10734 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 171, Last known committed idx: 170, Time since last communication: 0.000s
I20240501 19:56:27.875247 10510 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 123 to 171, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:27.878623 8693 catalog_manager.cc:5484] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a reported cstate change: config changed from index 123 to 171, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:27.879272 10121 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Committing config change with OpId 1.171: config changed from index 123 to 171, c9564f3b39b549c1bd995354fa135c5f (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:27.895300 8839 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3"
dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60070
I20240501 19:56:27.895485 8839 raft_consensus.cc:595] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:27.899621 10252 raft_consensus.cc:1203] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.110->[1.111-1.143] Dedup: 1.133->[1.134-1.143]
W20240501 19:56:27.904796 8799 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42240: Service unavailable: leader transfer in progress
W20240501 19:56:27.912863 8799 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42240: Service unavailable: leader transfer in progress
W20240501 19:56:27.919925 8799 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42240: Service unavailable: leader transfer in progress
W20240501 19:56:27.925542 8799 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42240: Service unavailable: leader transfer in progress
I20240501 19:56:27.928236 11257 raft_consensus.cc:1050] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: attempting to promote NON_VOTER 6977f9a52e9b4d06bcb12f9bee61553b to VOTER
I20240501 19:56:27.928593 11257 consensus_queue.cc:237] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 145, Committed index: 145, Last appended: 1.145, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:27.931221 10649 raft_consensus.cc:979] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: : Instructing follower 6977f9a52e9b4d06bcb12f9bee61553b to start an election
I20240501 19:56:27.931252 10249 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 LEARNER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch)
I20240501 19:56:27.931305 10649 raft_consensus.cc:1067] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Signalling peer 6977f9a52e9b4d06bcb12f9bee61553b to start an election
I20240501 19:56:27.931910 10649 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s
I20240501 19:56:27.932229 10249 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3"
dest_uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
from {username='slave'} at 127.5.139.65:37941
I20240501 19:56:27.932404 10249 raft_consensus.cc:482] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:27.932525 10249 raft_consensus.cc:3037] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:27.933710 10119 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Refusing update from remote peer 0f2b49c6d3dc49fabee40802e73ac982: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 1 index: 146. (index mismatch)
I20240501 19:56:27.933730 10249 raft_consensus.cc:504] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:27.934679 9923 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s
I20240501 19:56:27.935071 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3" candidate_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" candidate_term: 2 candidate_status { last_received { term: 1 index: 150 } } ignore_live_leader: true dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
I20240501 19:56:27.935202 8839 raft_consensus.cc:3032] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:27.935251 8839 raft_consensus.cc:729] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Running, Role: LEADER
I20240501 19:56:27.935395 8839 consensus_queue.cc:260] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 150, Committed index: 150, Last appended: 1.150, Last appended by leader: 150, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:27.935523 8839 raft_consensus.cc:3037] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:27.935628 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42240: Illegal state: replica 0f2b49c6d3dc49fabee40802e73ac982 is not leader of this config: current role FOLLOWER
I20240501 19:56:27.936429 8839 raft_consensus.cc:2454] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6977f9a52e9b4d06bcb12f9bee61553b in term 2.
I20240501 19:56:27.936780 10185 leader_election.cc:302] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, 6977f9a52e9b4d06bcb12f9bee61553b; no voters:
I20240501 19:56:27.937108 10382 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3" candidate_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" candidate_term: 2 candidate_status { last_received { term: 1 index: 150 } } ignore_live_leader: true dest_uuid: "6b4f42453a864314adc86aba2c3b3556"
I20240501 19:56:27.937266 10382 raft_consensus.cc:3037] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:27.937639 11229 raft_consensus.cc:2781] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:27.937799 11229 raft_consensus.cc:686] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Becoming Leader. State: Replica: 6977f9a52e9b4d06bcb12f9bee61553b, State: Running, Role: LEADER
I20240501 19:56:27.938077 11229 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 150, Committed index: 150, Last appended: 1.150, Last appended by leader: 150, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:27.938258 10382 raft_consensus.cc:2454] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6977f9a52e9b4d06bcb12f9bee61553b in term 2.
I20240501 19:56:27.940960 10654 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Committing config change with OpId 1.146: config changed from index 111 to 146, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:27.941221 8693 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b reported cstate change: term changed from 1 to 2, leader changed from 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) to 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69). New cstate: current_term: 2 leader_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" committed_config { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:27.943581 8694 catalog_manager.cc:5484] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 reported cstate change: config changed from index 111 to 146, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "0f2b49c6d3dc49fabee40802e73ac982" committed_config { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:27.944453 10249 leader_election.cc:288] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [CANDIDATE]: Term 2 election: Requested vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731), 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441)
I20240501 19:56:27.948840 10120 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 111 to 146, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:27.956048 10252 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.146: config changed from index 111 to 146, 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:27.980783 9101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49f4147edb7349bcbf51994e59501d19"
dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49056
I20240501 19:56:27.980927 9101 raft_consensus.cc:595] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:27.982993 9902 raft_consensus.cc:979] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: : Instructing follower 14798909ff724444a8be03c45c41929a to start an election
I20240501 19:56:27.983090 9902 raft_consensus.cc:1067] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Signalling peer 14798909ff724444a8be03c45c41929a to start an election
I20240501 19:56:27.984699 10510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49f4147edb7349bcbf51994e59501d19"
dest_uuid: "14798909ff724444a8be03c45c41929a"
from {username='slave'} at 127.5.139.67:34587
I20240501 19:56:27.984880 10510 raft_consensus.cc:482] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:27.984943 10510 raft_consensus.cc:3037] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:27.985940 10510 raft_consensus.cc:504] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:27.986459 10510 leader_election.cc:288] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 2 election: Requested vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389), c9564f3b39b549c1bd995354fa135c5f (127.5.139.68:46413)
I20240501 19:56:27.987656 8837 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Refusing update from remote peer 6977f9a52e9b4d06bcb12f9bee61553b: Log matching property violated. Preceding OpId in replica: term: 1 index: 150. Preceding OpId from leader: term: 2 index: 152. (index mismatch)
I20240501 19:56:27.988294 11301 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240501 19:56:27.988638 10382 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Refusing update from remote peer 6977f9a52e9b4d06bcb12f9bee61553b: Log matching property violated. Preceding OpId in replica: term: 1 index: 150. Preceding OpId from leader: term: 2 index: 152. (index mismatch)
I20240501 19:56:27.989264 11301 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240501 19:56:27.993016 10119 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f4147edb7349bcbf51994e59501d19" candidate_uuid: "14798909ff724444a8be03c45c41929a" candidate_term: 2 candidate_status { last_received { term: 1 index: 177 } } ignore_live_leader: true dest_uuid: "c9564f3b39b549c1bd995354fa135c5f"
I20240501 19:56:27.993299 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49f4147edb7349bcbf51994e59501d19" candidate_uuid: "14798909ff724444a8be03c45c41929a" candidate_term: 2 candidate_status { last_received { term: 1 index: 177 } } ignore_live_leader: true dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
I20240501 19:56:27.993409 9101 raft_consensus.cc:3032] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:27.993461 9101 raft_consensus.cc:729] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:27.993602 9101 consensus_queue.cc:260] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 1.177, Last appended by leader: 177, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:27.993785 9101 raft_consensus.cc:3037] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Advancing to term 2
W20240501 19:56:27.993420 9060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44994: Service unavailable: leader transfer in progress
I20240501 19:56:27.994859 10119 raft_consensus.cc:3037] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:27.994894 9101 raft_consensus.cc:2454] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14798909ff724444a8be03c45c41929a in term 2.
I20240501 19:56:27.995209 10450 leader_election.cc:302] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14798909ff724444a8be03c45c41929a, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:27.995407 11244 raft_consensus.cc:2781] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:27.995494 11244 raft_consensus.cc:686] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 2 LEADER]: Becoming Leader. State: Replica: 14798909ff724444a8be03c45c41929a, State: Running, Role: LEADER
I20240501 19:56:27.995648 11244 consensus_queue.cc:237] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 1.177, Last appended by leader: 177, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
W20240501 19:56:27.996989 9060 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:44994: Illegal state: replica b3b45c9058d84b4e9ace42c9154ab0e0 is not leader of this config: current role FOLLOWER
I20240501 19:56:27.997084 10958 mvcc.cc:205] Tried to move back new op lower bound from 7022974517192204288 to 7022974516997246976. Current Snapshot: MvccSnapshot[applied={T|T < 7022974517192204288 or (T in {7022974517192204288})}]
I20240501 19:56:27.997167 8693 catalog_manager.cc:5484] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a reported cstate change: term changed from 1 to 2, leader changed from b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) to 14798909ff724444a8be03c45c41929a (127.5.139.71). New cstate: current_term: 2 leader_uuid: "14798909ff724444a8be03c45c41929a" committed_config { opid_index: 171 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:27.997334 10119 raft_consensus.cc:2454] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14798909ff724444a8be03c45c41929a in term 2.
I20240501 19:56:28.045071 10515 raft_consensus.cc:1203] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.105->[1.106-1.149] Dedup: 1.126->[1.127-1.149]
I20240501 19:56:28.052664 10249 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 2.155, Last appended by leader: 150, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } }
I20240501 19:56:28.056823 10382 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Refusing update from remote peer 6977f9a52e9b4d06bcb12f9bee61553b: Log matching property violated. Preceding OpId in replica: term: 2 index: 155. Preceding OpId from leader: term: 2 index: 156. (index mismatch)
I20240501 19:56:28.057286 11301 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240501 19:56:28.061630 11301 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Committing config change with OpId 2.156: config changed from index 146 to 156, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:28.063999 8681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d827d0b0d5ba4969908768c96f92c1f3 with cas_config_opid_index 146: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:28.064884 10382 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Committing config change with OpId 2.156: config changed from index 146 to 156, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } }
I20240501 19:56:28.066972 8694 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b reported cstate change: config changed from index 146 to 156, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New cstate: current_term: 2 leader_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" committed_config { opid_index: 156 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:28.070098 8819 tablet_service.cc:1514] Processing DeleteTablet for tablet d827d0b0d5ba4969908768c96f92c1f3 with delete_type TABLET_DATA_TOMBSTONED (TS 0f2b49c6d3dc49fabee40802e73ac982 not found in new config with opid_index 156) from {username='slave'} at 127.0.0.1:48934
I20240501 19:56:28.070323 11264 tablet_replica.cc:330] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: stopping tablet replica
I20240501 19:56:28.070456 11264 raft_consensus.cc:2229] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:28.070586 11264 raft_consensus.cc:2258] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:28.071632 11264 ts_tablet_manager.cc:1875] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:28.074020 11264 ts_tablet_manager.cc:1888] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.155
I20240501 19:56:28.074095 11264 log.cc:1196] T d827d0b0d5ba4969908768c96f92c1f3 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:28.075675 8681 catalog_manager.cc:4830] TS 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731): tablet d827d0b0d5ba4969908768c96f92c1f3 (table rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]) successfully deleted
I20240501 19:56:28.082389 9101 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 1 index: 177. Preceding OpId from leader: term: 2 index: 179. (index mismatch)
I20240501 19:56:28.082473 10119 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 1 index: 177. Preceding OpId from leader: term: 2 index: 179. (index mismatch)
I20240501 19:56:28.083127 11244 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I20240501 19:56:28.083389 11200 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I20240501 19:56:28.085850 11289 mvcc.cc:205] Tried to move back new op lower bound from 7022974517581795328 to 7022974517231669248. Current Snapshot: MvccSnapshot[applied={T|T < 7022974517581795328}]
I20240501 19:56:28.099272 10510 consensus_queue.cc:237] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 179, Committed index: 179, Last appended: 2.179, Last appended by leader: 177, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } }
I20240501 19:56:28.102523 10119 raft_consensus.cc:1261] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 2 index: 179. Preceding OpId from leader: term: 2 index: 180. (index mismatch)
I20240501 19:56:28.103202 11200 consensus_queue.cc:1035] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 180, Last known committed idx: 179, Time since last communication: 0.000s
I20240501 19:56:28.105427 11200 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a [term 2 LEADER]: Committing config change with OpId 2.180: config changed from index 171 to 180, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:28.105880 10121 raft_consensus.cc:2932] T 49f4147edb7349bcbf51994e59501d19 P c9564f3b39b549c1bd995354fa135c5f [term 2 FOLLOWER]: Committing config change with OpId 2.180: config changed from index 171 to 180, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } }
I20240501 19:56:28.107017 8682 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49f4147edb7349bcbf51994e59501d19 with cas_config_opid_index 171: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:28.108178 8693 catalog_manager.cc:5484] T 49f4147edb7349bcbf51994e59501d19 P 14798909ff724444a8be03c45c41929a reported cstate change: config changed from index 171 to 180, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New cstate: current_term: 2 leader_uuid: "14798909ff724444a8be03c45c41929a" committed_config { opid_index: 180 OBSOLETE_local: false peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:28.112242 9080 tablet_service.cc:1514] Processing DeleteTablet for tablet 49f4147edb7349bcbf51994e59501d19 with delete_type TABLET_DATA_TOMBSTONED (TS b3b45c9058d84b4e9ace42c9154ab0e0 not found in new config with opid_index 180) from {username='slave'} at 127.0.0.1:44930
I20240501 19:56:28.113842 11321 tablet_replica.cc:330] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: stopping tablet replica
I20240501 19:56:28.113997 11321 raft_consensus.cc:2229] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:28.114162 11321 raft_consensus.cc:2258] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:28.115285 11321 ts_tablet_manager.cc:1875] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:28.117992 11321 ts_tablet_manager.cc:1888] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.179
I20240501 19:56:28.118139 11321 log.cc:1196] T 49f4147edb7349bcbf51994e59501d19 P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/49f4147edb7349bcbf51994e59501d19
I20240501 19:56:28.118904 8684 catalog_manager.cc:4830] TS b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389): tablet 49f4147edb7349bcbf51994e59501d19 (table rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]) successfully deleted
I20240501 19:56:28.122660 9986 raft_consensus.cc:1050] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: attempting to promote NON_VOTER 14798909ff724444a8be03c45c41929a to VOTER
I20240501 19:56:28.123003 9986 consensus_queue.cc:237] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 150, Committed index: 150, Last appended: 1.150, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } }
I20240501 19:56:28.124104 10382 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 150. Preceding OpId from leader: term: 1 index: 151. (index mismatch)
I20240501 19:56:28.124614 11008 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240501 19:56:28.126703 10510 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 1 LEARNER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 150. Preceding OpId from leader: term: 1 index: 151. (index mismatch)
I20240501 19:56:28.127373 10615 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.151: config changed from index 106 to 151, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:28.127965 11138 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 151, Last known committed idx: 150, Time since last communication: 0.000s
I20240501 19:56:28.128871 10510 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Committing config change with OpId 1.151: config changed from index 106 to 151, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:28.129621 10382 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Committing config change with OpId 1.151: config changed from index 106 to 151, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:28.132369 8693 catalog_manager.cc:5484] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index 106 to 151, 14798909ff724444a8be03c45c41929a (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:28.209581 8839 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df35daed2f72400086ccd0b8c47ecab1"
dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60078
I20240501 19:56:28.209731 8839 raft_consensus.cc:595] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Received request to transfer leadership
W20240501 19:56:28.247174 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.257513 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.264570 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.273814 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.281998 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.289865 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
I20240501 19:56:28.291918 8969 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68c8311312bd4b50bdbedf414ada6bd2"
dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57668
I20240501 19:56:28.292109 8969 raft_consensus.cc:595] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Received request to transfer leadership
W20240501 19:56:28.292912 10988 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.296681 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.299598 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.303786 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.310469 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.311210 8927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.316485 8927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.323645 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.325768 8927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
I20240501 19:56:28.334105 8839 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df35daed2f72400086ccd0b8c47ecab1"
dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60092
I20240501 19:56:28.334223 8839 raft_consensus.cc:595] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Received request to transfer leadership
W20240501 19:56:28.337507 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.339164 8927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.350282 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.352804 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.363559 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.365808 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.376746 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.381302 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
I20240501 19:56:28.386948 8969 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68c8311312bd4b50bdbedf414ada6bd2"
dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57692
I20240501 19:56:28.387074 8969 raft_consensus.cc:595] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Received request to transfer leadership
W20240501 19:56:28.392024 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.399310 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.406332 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.418660 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.424749 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.439148 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.439213 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.458408 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.459757 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.476662 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.479744 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.500602 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.503124 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.522632 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.527981 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.546123 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.549777 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.567633 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.572926 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.588930 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.598690 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
I20240501 19:56:28.606417 8839 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "df35daed2f72400086ccd0b8c47ecab1"
dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60096
I20240501 19:56:28.606559 8839 raft_consensus.cc:595] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Received request to transfer leadership
W20240501 19:56:28.613359 8927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
I20240501 19:56:28.622653 8969 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "68c8311312bd4b50bdbedf414ada6bd2"
dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57704
I20240501 19:56:28.622795 8969 raft_consensus.cc:595] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Received request to transfer leadership
W20240501 19:56:28.625876 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.636902 8927 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.654117 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.665180 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.682927 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Service unavailable: leader transfer in progress
W20240501 19:56:28.690563 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
I20240501 19:56:28.698505 9923 raft_consensus.cc:979] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: : Instructing follower c9564f3b39b549c1bd995354fa135c5f to start an election
I20240501 19:56:28.698630 9923 raft_consensus.cc:1067] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Signalling peer c9564f3b39b549c1bd995354fa135c5f to start an election
I20240501 19:56:28.699126 10119 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "df35daed2f72400086ccd0b8c47ecab1"
dest_uuid: "c9564f3b39b549c1bd995354fa135c5f"
from {username='slave'} at 127.5.139.65:54743
I20240501 19:56:28.699303 10119 raft_consensus.cc:482] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:28.699368 10119 raft_consensus.cc:3037] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.700511 10119 raft_consensus.cc:504] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:28.702791 8839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df35daed2f72400086ccd0b8c47ecab1" candidate_uuid: "c9564f3b39b549c1bd995354fa135c5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: true dest_uuid: "0f2b49c6d3dc49fabee40802e73ac982"
I20240501 19:56:28.702972 8839 raft_consensus.cc:3032] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:28.703120 8839 raft_consensus.cc:729] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0f2b49c6d3dc49fabee40802e73ac982, State: Running, Role: LEADER
I20240501 19:56:28.703258 10119 leader_election.cc:288] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [CANDIDATE]: Term 2 election: Requested vote from peers 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731), 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483)
I20240501 19:56:28.703485 8839 consensus_queue.cc:260] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 1.160, Last appended by leader: 160, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:28.704118 8839 raft_consensus.cc:3037] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.705350 8839 raft_consensus.cc:2454] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9564f3b39b549c1bd995354fa135c5f in term 2.
I20240501 19:56:28.705727 10054 leader_election.cc:302] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f2b49c6d3dc49fabee40802e73ac982, c9564f3b39b549c1bd995354fa135c5f; no voters:
I20240501 19:56:28.707183 11394 raft_consensus.cc:2781] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:28.707418 11394 raft_consensus.cc:686] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 2 LEADER]: Becoming Leader. State: Replica: c9564f3b39b549c1bd995354fa135c5f, State: Running, Role: LEADER
I20240501 19:56:28.707844 11394 consensus_queue.cc:237] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 160, Committed index: 160, Last appended: 1.160, Last appended by leader: 160, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:28.708213 10251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df35daed2f72400086ccd0b8c47ecab1" candidate_uuid: "c9564f3b39b549c1bd995354fa135c5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 160 } } ignore_live_leader: true dest_uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
I20240501 19:56:28.708361 10251 raft_consensus.cc:3037] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.710317 10251 raft_consensus.cc:2454] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9564f3b39b549c1bd995354fa135c5f in term 2.
I20240501 19:56:28.711601 8694 catalog_manager.cc:5484] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f reported cstate change: term changed from 1 to 2, leader changed from 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) to c9564f3b39b549c1bd995354fa135c5f (127.5.139.68). New cstate: current_term: 2 leader_uuid: "c9564f3b39b549c1bd995354fa135c5f" committed_config { opid_index: 146 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:28.714105 8797 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:42218: Illegal state: replica 0f2b49c6d3dc49fabee40802e73ac982 is not leader of this config: current role FOLLOWER
W20240501 19:56:28.719614 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
W20240501 19:56:28.749986 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
I20240501 19:56:28.781088 11008 raft_consensus.cc:979] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: : Instructing follower 14798909ff724444a8be03c45c41929a to start an election
I20240501 19:56:28.781198 11008 raft_consensus.cc:1067] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Signalling peer 14798909ff724444a8be03c45c41929a to start an election
I20240501 19:56:28.781701 10510 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "68c8311312bd4b50bdbedf414ada6bd2"
dest_uuid: "14798909ff724444a8be03c45c41929a"
from {username='slave'} at 127.5.139.66:44031
W20240501 19:56:28.782497 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Service unavailable: leader transfer in progress
I20240501 19:56:28.783505 10510 raft_consensus.cc:482] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:28.783591 10510 raft_consensus.cc:3037] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.785248 10510 raft_consensus.cc:504] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } }
I20240501 19:56:28.785916 8969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c8311312bd4b50bdbedf414ada6bd2" candidate_uuid: "14798909ff724444a8be03c45c41929a" candidate_term: 2 candidate_status { last_received { term: 1 index: 155 } } ignore_live_leader: true dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
I20240501 19:56:28.786021 8969 raft_consensus.cc:3032] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:28.786083 8969 raft_consensus.cc:729] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Running, Role: LEADER
I20240501 19:56:28.786240 8969 consensus_queue.cc:260] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 1.155, Last appended by leader: 155, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } }
I20240501 19:56:28.786413 8969 raft_consensus.cc:3037] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.787159 8969 raft_consensus.cc:2454] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14798909ff724444a8be03c45c41929a in term 2.
I20240501 19:56:28.787432 10448 leader_election.cc:302] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14798909ff724444a8be03c45c41929a, ac13bdb319c448dba072fb9e62d6893b; no voters:
I20240501 19:56:28.787628 11320 raft_consensus.cc:2781] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:28.787740 11320 raft_consensus.cc:686] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 2 LEADER]: Becoming Leader. State: Replica: 14798909ff724444a8be03c45c41929a, State: Running, Role: LEADER
I20240501 19:56:28.787859 11320 consensus_queue.cc:237] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 155, Committed index: 155, Last appended: 1.155, Last appended by leader: 155, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } }
I20240501 19:56:28.790231 10510 leader_election.cc:288] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [CANDIDATE]: Term 2 election: Requested vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725), 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441)
I20240501 19:56:28.790692 8694 catalog_manager.cc:5484] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a reported cstate change: term changed from 1 to 2, leader changed from ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) to 14798909ff724444a8be03c45c41929a (127.5.139.71). New cstate: current_term: 2 leader_uuid: "14798909ff724444a8be03c45c41929a" committed_config { opid_index: 151 OBSOLETE_local: false peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:28.799434 10251 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Refusing update from remote peer c9564f3b39b549c1bd995354fa135c5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 2 index: 162. (index mismatch)
I20240501 19:56:28.799716 8839 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Refusing update from remote peer c9564f3b39b549c1bd995354fa135c5f: Log matching property violated. Preceding OpId in replica: term: 1 index: 160. Preceding OpId from leader: term: 2 index: 162. (index mismatch)
I20240501 19:56:28.800164 11394 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240501 19:56:28.800585 11395 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 161, Last known committed idx: 160, Time since last communication: 0.000s
I20240501 19:56:28.804309 11271 mvcc.cc:205] Tried to move back new op lower bound from 7022974520518508544 to 7022974520151740416. Current Snapshot: MvccSnapshot[applied={T|T < 7022974520518508544 or (T in {7022974520518508544})}]
I20240501 19:56:28.805809 10381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68c8311312bd4b50bdbedf414ada6bd2" candidate_uuid: "14798909ff724444a8be03c45c41929a" candidate_term: 2 candidate_status { last_received { term: 1 index: 155 } } ignore_live_leader: true dest_uuid: "6b4f42453a864314adc86aba2c3b3556"
I20240501 19:56:28.805909 10381 raft_consensus.cc:3037] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:28.806882 10381 raft_consensus.cc:2454] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14798909ff724444a8be03c45c41929a in term 2.
I20240501 19:56:28.809203 11228 mvcc.cc:205] Tried to move back new op lower bound from 7022974520518508544 to 7022974520151740416. Current Snapshot: MvccSnapshot[applied={T|T < 7022974520518508544 or (T in {7022974520518508544})}]
W20240501 19:56:28.812139 8930 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:45702: Illegal state: replica ac13bdb319c448dba072fb9e62d6893b is not leader of this config: current role FOLLOWER
I20240501 19:56:28.827927 10119 consensus_queue.cc:237] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 163, Committed index: 163, Last appended: 2.164, Last appended by leader: 160, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } }
I20240501 19:56:28.830675 10251 raft_consensus.cc:1261] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Refusing update from remote peer c9564f3b39b549c1bd995354fa135c5f: Log matching property violated. Preceding OpId in replica: term: 2 index: 164. Preceding OpId from leader: term: 2 index: 165. (index mismatch)
I20240501 19:56:28.831110 11406 consensus_queue.cc:1035] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 165, Last known committed idx: 163, Time since last communication: 0.000s
I20240501 19:56:28.843358 11406 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f [term 2 LEADER]: Committing config change with OpId 2.165: config changed from index 146 to 165, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:28.845291 8681 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet df35daed2f72400086ccd0b8c47ecab1 with cas_config_opid_index 146: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:28.847059 8694 catalog_manager.cc:5484] T df35daed2f72400086ccd0b8c47ecab1 P c9564f3b39b549c1bd995354fa135c5f reported cstate change: config changed from index 146 to 165, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New cstate: current_term: 2 leader_uuid: "c9564f3b39b549c1bd995354fa135c5f" committed_config { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:28.852829 10251 raft_consensus.cc:2932] T df35daed2f72400086ccd0b8c47ecab1 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Committing config change with OpId 2.165: config changed from index 146 to 165, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 165 OBSOLETE_local: false peers { permanent_uuid: "c9564f3b39b549c1bd995354fa135c5f" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 46413 } attrs { promote: false } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } }
I20240501 19:56:28.853384 8819 tablet_service.cc:1514] Processing DeleteTablet for tablet df35daed2f72400086ccd0b8c47ecab1 with delete_type TABLET_DATA_TOMBSTONED (TS 0f2b49c6d3dc49fabee40802e73ac982 not found in new config with opid_index 165) from {username='slave'} at 127.0.0.1:48934
I20240501 19:56:28.853708 11421 tablet_replica.cc:330] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: stopping tablet replica
I20240501 19:56:28.853931 11421 raft_consensus.cc:2229] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:28.854116 11421 pending_rounds.cc:62] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Trying to abort 1 pending ops.
I20240501 19:56:28.854202 11421 pending_rounds.cc:68] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Aborting op as it isn't in flight: id { term: 2 index: 164 } timestamp: 7022974520615489536 op_type: WRITE_OP write_request { tablet_id: "df35daed2f72400086ccd0b8c47ecab1" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\261\205T\007\006\313\310\337\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""26d1f2eac3462ac5" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022974520595984384 authz_token { token_data: "\010\223\277\312\261\006\"3\n\005slave\022*\n 3c90b69a201147fa8697594a2940e3a2\020\001\030\001 \001(\001" signature: "<redacted>""\022\244\0312\373Z\226*\020\222/\315\262\232\0244\315\276^1\\h\266;t\340xR\024\351\037\335]\205\346|EJ\254ev\335\340\327\211\017a~\350\035\376\353\311y\365BG\017\355\"\330\302\251\267" signing_key_seq_num: 0 } } request_id { client_id: "d64764e15e6c43cb8a5c3b0f7d1181c4" seq_no: 461 first_incomplete_seq_no: 461 attempt_no: 0 }
I20240501 19:56:28.854816 11421 raft_consensus.cc:2258] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:28.855816 11421 ts_tablet_manager.cc:1875] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:28.858595 11421 ts_tablet_manager.cc:1888] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.164
I20240501 19:56:28.858729 11421 log.cc:1196] T df35daed2f72400086ccd0b8c47ecab1 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/df35daed2f72400086ccd0b8c47ecab1
I20240501 19:56:28.872233 8681 catalog_manager.cc:4830] TS 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731): tablet df35daed2f72400086ccd0b8c47ecab1 (table rebalance_test_table_1 [id=3c90b69a201147fa8697594a2940e3a2]) successfully deleted
I20240501 19:56:28.920536 10382 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 2 index: 157. (index mismatch)
I20240501 19:56:28.921010 11244 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240501 19:56:28.921864 8969 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 1 index: 155. Preceding OpId from leader: term: 2 index: 157. (index mismatch)
I20240501 19:56:28.923318 11320 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 156, Last known committed idx: 155, Time since last communication: 0.000s
I20240501 19:56:28.931926 10515 consensus_queue.cc:237] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 157, Committed index: 157, Last appended: 2.157, Last appended by leader: 155, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } }
I20240501 19:56:28.935814 10382 raft_consensus.cc:1261] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 2 index: 157. Preceding OpId from leader: term: 2 index: 158. (index mismatch)
I20240501 19:56:28.936218 11320 consensus_queue.cc:1035] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 158, Last known committed idx: 157, Time since last communication: 0.000s
I20240501 19:56:28.938165 11320 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P 14798909ff724444a8be03c45c41929a [term 2 LEADER]: Committing config change with OpId 2.158: config changed from index 151 to 158, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:28.938642 10381 raft_consensus.cc:2932] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Committing config change with OpId 2.158: config changed from index 151 to 158, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:28.939733 8682 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 68c8311312bd4b50bdbedf414ada6bd2 with cas_config_opid_index 151: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:28.942010 8694 catalog_manager.cc:5484] T 68c8311312bd4b50bdbedf414ada6bd2 P 6b4f42453a864314adc86aba2c3b3556 reported cstate change: config changed from index 151 to 158, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New cstate: current_term: 2 leader_uuid: "14798909ff724444a8be03c45c41929a" committed_config { opid_index: 158 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } attrs { promote: false } } }
I20240501 19:56:28.944751 8949 tablet_service.cc:1514] Processing DeleteTablet for tablet 68c8311312bd4b50bdbedf414ada6bd2 with delete_type TABLET_DATA_TOMBSTONED (TS ac13bdb319c448dba072fb9e62d6893b not found in new config with opid_index 158) from {username='slave'} at 127.0.0.1:45626
I20240501 19:56:28.945748 11425 tablet_replica.cc:330] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: stopping tablet replica
I20240501 19:56:28.945917 11425 raft_consensus.cc:2229] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:28.946570 11425 raft_consensus.cc:2258] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:28.954078 11425 ts_tablet_manager.cc:1875] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:28.956895 11425 ts_tablet_manager.cc:1888] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.157
I20240501 19:56:28.956974 11425 log.cc:1196] T 68c8311312bd4b50bdbedf414ada6bd2 P ac13bdb319c448dba072fb9e62d6893b: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/68c8311312bd4b50bdbedf414ada6bd2
I20240501 19:56:28.958516 8682 catalog_manager.cc:4830] TS ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725): tablet 68c8311312bd4b50bdbedf414ada6bd2 (table rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]) successfully deleted
I20240501 19:56:29.403681 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:29.422878 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:29.434384 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:29.491894 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:29.501669 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:29.512812 10232 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:29.521277 8949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:29.790632 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:29.798269 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:29.800935 10232 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:29.801470 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:29.843123 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:29.849239 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:29.851707 8949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:30.627964 8949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:30.637204 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:30.647311 10232 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:30.651857 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:30.652001 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:30.664238 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:30.664707 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:30.988493 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:30.989457 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:31.003377 8949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:31.009183 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:31.012159 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:31.033792 10232 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:31.066891 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:31.243665 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:31.245591 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:31.247536 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:31.250849 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:31.255437 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:31.256078 8949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:31.261386 10232 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
2ee3823193724caab53a0a77e70e18df | 127.5.139.126:34427 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.139.84:42171 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0f2b49c6d3dc49fabee40802e73ac982 | 127.5.139.65:38731 | HEALTHY | <none> | 3 | 0
14798909ff724444a8be03c45c41929a | 127.5.139.71:38393 | HEALTHY | <none> | 3 | 0
6977f9a52e9b4d06bcb12f9bee61553b | 127.5.139.69:42483 | HEALTHY | <none> | 1 | 0
6b4f42453a864314adc86aba2c3b3556 | 127.5.139.70:43441 | HEALTHY | <none> | 1 | 0
ac13bdb319c448dba072fb9e62d6893b | 127.5.139.66:43725 | HEALTHY | <none> | 3 | 0
b3b45c9058d84b4e9ace42c9154ab0e0 | 127.5.139.67:35389 | HEALTHY | <none> | 4 | 0
c9564f3b39b549c1bd995354fa135c5f | 127.5.139.68:46413 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.5.139.65 | experimental | 127.5.139.65:38731
local_ip_for_outbound_sockets | 127.5.139.66 | experimental | 127.5.139.66:43725
local_ip_for_outbound_sockets | 127.5.139.67 | experimental | 127.5.139.67:35389
local_ip_for_outbound_sockets | 127.5.139.68 | experimental | 127.5.139.68:46413
local_ip_for_outbound_sockets | 127.5.139.69 | experimental | 127.5.139.69:42483
local_ip_for_outbound_sockets | 127.5.139.70 | experimental | 127.5.139.70:43441
local_ip_for_outbound_sockets | 127.5.139.71 | experimental | 127.5.139.71:38393
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.139.65:38731
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.139.66:43725
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.139.67:35389
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.139.68:46413
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.139.69:42483
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.139.70:43441
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.139.71:38393
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.139.84:42171 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 19:56:31.292997 5677 log_verifier.cc:126] Checking tablet 0c4e313b03cc44d0a2f7864f7e9d4f90
I20240501 19:56:31.345683 5677 log_verifier.cc:177] Verified matching terms for 277 ops in tablet 0c4e313b03cc44d0a2f7864f7e9d4f90
I20240501 19:56:31.345805 5677 log_verifier.cc:126] Checking tablet 0ca216ce0c6a420f8c90273a02ad8cf4
I20240501 19:56:31.398344 5677 log_verifier.cc:177] Verified matching terms for 257 ops in tablet 0ca216ce0c6a420f8c90273a02ad8cf4
I20240501 19:56:31.398471 5677 log_verifier.cc:126] Checking tablet 0f1ab4dc3bb841c2b352cee4ec7aa373
I20240501 19:56:31.448822 5677 log_verifier.cc:177] Verified matching terms for 258 ops in tablet 0f1ab4dc3bb841c2b352cee4ec7aa373
I20240501 19:56:31.448951 5677 log_verifier.cc:126] Checking tablet 3211e49e29174e0dbf667959a78c30e2
I20240501 19:56:31.493594 5677 log_verifier.cc:177] Verified matching terms for 221 ops in tablet 3211e49e29174e0dbf667959a78c30e2
I20240501 19:56:31.493698 5677 log_verifier.cc:126] Checking tablet 49f4147edb7349bcbf51994e59501d19
I20240501 19:56:31.553896 5677 log_verifier.cc:177] Verified matching terms for 322 ops in tablet 49f4147edb7349bcbf51994e59501d19
I20240501 19:56:31.554061 5677 log_verifier.cc:126] Checking tablet 58979b7e47734549bc48710ff1d2b1c1
I20240501 19:56:31.609102 5677 log_verifier.cc:177] Verified matching terms for 264 ops in tablet 58979b7e47734549bc48710ff1d2b1c1
I20240501 19:56:31.609216 5677 log_verifier.cc:126] Checking tablet 68c8311312bd4b50bdbedf414ada6bd2
I20240501 19:56:31.654955 5677 log_verifier.cc:177] Verified matching terms for 230 ops in tablet 68c8311312bd4b50bdbedf414ada6bd2
I20240501 19:56:31.655063 5677 log_verifier.cc:126] Checking tablet 6db903f127ce471d9cb87537a3cb7286
I20240501 19:56:31.706554 5677 log_verifier.cc:177] Verified matching terms for 246 ops in tablet 6db903f127ce471d9cb87537a3cb7286
I20240501 19:56:31.706665 5677 log_verifier.cc:126] Checking tablet 74706b357b2b43549757d24fa9a9d282
I20240501 19:56:31.758539 5677 log_verifier.cc:177] Verified matching terms for 270 ops in tablet 74706b357b2b43549757d24fa9a9d282
I20240501 19:56:31.758658 5677 log_verifier.cc:126] Checking tablet aa477b2bd34d4b74b245314148e4851d
I20240501 19:56:31.811432 5677 log_verifier.cc:177] Verified matching terms for 279 ops in tablet aa477b2bd34d4b74b245314148e4851d
I20240501 19:56:31.811558 5677 log_verifier.cc:126] Checking tablet ac9114fe81be4ba0bb4aadf28c61490d
I20240501 19:56:31.865689 5677 log_verifier.cc:177] Verified matching terms for 283 ops in tablet ac9114fe81be4ba0bb4aadf28c61490d
I20240501 19:56:31.865847 5677 log_verifier.cc:126] Checking tablet c63e353d2a544ce0af8551d8024743b7
I20240501 19:56:31.914243 5677 log_verifier.cc:177] Verified matching terms for 249 ops in tablet c63e353d2a544ce0af8551d8024743b7
I20240501 19:56:31.914372 5677 log_verifier.cc:126] Checking tablet c6e64c2bfdce4caf9c91618400c7f5d6
I20240501 19:56:31.928174 5677 log_verifier.cc:177] Verified matching terms for 2 ops in tablet c6e64c2bfdce4caf9c91618400c7f5d6
I20240501 19:56:31.928251 5677 log_verifier.cc:126] Checking tablet c869ad944b7a45d39ec171b0add850fb
I20240501 19:56:31.973987 5677 log_verifier.cc:177] Verified matching terms for 225 ops in tablet c869ad944b7a45d39ec171b0add850fb
I20240501 19:56:31.974115 5677 log_verifier.cc:126] Checking tablet d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:32.026845 5677 log_verifier.cc:177] Verified matching terms for 274 ops in tablet d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:32.026988 5677 log_verifier.cc:126] Checking tablet df35daed2f72400086ccd0b8c47ecab1
I20240501 19:56:32.073083 5677 log_verifier.cc:177] Verified matching terms for 224 ops in tablet df35daed2f72400086ccd0b8c47ecab1
I20240501 19:56:32.075057 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.72:0
--local_ip_for_outbound_sockets=127.5.139.72
--webserver_interface=127.5.139.72
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--builtin_ntp_servers=127.5.139.84:42171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:32.187708 11650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:32.187906 11650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:32.187954 11650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:32.191171 11650 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:32.191231 11650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:32.191329 11650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.72
I20240501 19:56:32.194852 11650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:42171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.139.72
--webserver_port=0
--tserver_master_addrs=127.5.139.126:34427
--never_fsync=true
--heap_profile_path=/tmp/kudu.11650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.72
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:32.195657 11650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:32.196666 11650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:32.204876 11655 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:32.205497 11650 server_base.cc:1034] running on GCE node
W20240501 19:56:32.205847 11656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:32.206486 11658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:32.206848 11650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:32.207396 11650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:32.208551 11650 hybrid_clock.cc:648] HybridClock initialized: now 1714593392208535 us; error 31 us; skew 500 ppm
I20240501 19:56:32.210304 11650 webserver.cc:460] Webserver started at http://127.5.139.72:37181/ using document root <none> and password file <none>
I20240501 19:56:32.210840 11650 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:32.210901 11650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:32.211088 11650 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:32.212635 11650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "4663c01c9815427ab4bba05e1d1e44aa"
format_stamp: "Formatted at 2024-05-01 19:56:32 on dist-test-slave-twc7"
I20240501 19:56:32.213181 11650 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "4663c01c9815427ab4bba05e1d1e44aa"
format_stamp: "Formatted at 2024-05-01 19:56:32 on dist-test-slave-twc7"
I20240501 19:56:32.216406 11650 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:32.218499 11664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.219460 11650 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:32.219554 11650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "4663c01c9815427ab4bba05e1d1e44aa"
format_stamp: "Formatted at 2024-05-01 19:56:32 on dist-test-slave-twc7"
I20240501 19:56:32.219624 11650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:32.252265 11650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:32.252866 11650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:32.253026 11650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:32.253644 11650 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:32.254668 11650 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:32.254730 11650 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.254789 11650 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:32.254823 11650 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:32.263252 11650 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.72:33401
I20240501 19:56:32.263286 11777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.72:33401 every 8 connection(s)
I20240501 19:56:32.264214 11650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:32.271756 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 11650
I20240501 19:56:32.271893 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:56:32.272682 11778 heartbeater.cc:347] Connected to a master server at 127.5.139.126:34427
I20240501 19:56:32.272894 11778 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:32.273456 11778 heartbeater.cc:511] Master 127.5.139.126:34427 requested a full tablet report, sending...
I20240501 19:56:32.274377 8694 ts_manager.cc:194] Registered new tserver with Master: 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72:33401)
I20240501 19:56:32.274964 8694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.72:42657
I20240501 19:56:32.470485 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:32.472160 11712 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:32.475677 10232 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:32.478893 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:32.481268 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:32.482496 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:32.504829 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:32.512795 8949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:32.615553 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:32.618093 11712 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:32.625130 8949 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:32.626540 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:32.634224 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:32.636423 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:32.650367 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:32.657784 10232 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:32.708146 9101 consensus_queue.cc:237] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 279, Committed index: 279, Last appended: 1.279, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:32.709911 10252 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 279. Preceding OpId from leader: term: 1 index: 280. (index mismatch)
I20240501 19:56:32.710318 11874 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 280, Last known committed idx: 279, Time since last communication: 0.000s
I20240501 19:56:32.712440 11874 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.280: config changed from index 85 to 280, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
I20240501 19:56:32.713248 10252 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.280: config changed from index 85 to 280, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New config: { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
W20240501 19:56:32.714579 9036 consensus_peers.cc:485] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 -> Peer 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72:33401): Couldn't send request to peer 4663c01c9815427ab4bba05e1d1e44aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aa477b2bd34d4b74b245314148e4851d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:32.715955 8695 catalog_manager.cc:5484] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b reported cstate change: config changed from index 85 to 280, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
I20240501 19:56:32.725005 10252 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 274, Committed index: 274, Last appended: 2.274, Last appended by leader: 150, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:32.726614 10381 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Refusing update from remote peer 6977f9a52e9b4d06bcb12f9bee61553b: Log matching property violated. Preceding OpId in replica: term: 2 index: 274. Preceding OpId from leader: term: 2 index: 275. (index mismatch)
I20240501 19:56:32.727105 11888 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 275, Last known committed idx: 274, Time since last communication: 0.000s
I20240501 19:56:32.728860 11407 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Committing config change with OpId 2.275: config changed from index 156 to 275, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
I20240501 19:56:32.730585 10381 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Committing config change with OpId 2.275: config changed from index 156 to 275, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New config: { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
W20240501 19:56:32.733109 10187 consensus_peers.cc:485] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b -> Peer 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72:33401): Couldn't send request to peer 4663c01c9815427ab4bba05e1d1e44aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d827d0b0d5ba4969908768c96f92c1f3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:32.733898 8694 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 reported cstate change: config changed from index 156 to 275, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New cstate: current_term: 2 leader_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" committed_config { opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
I20240501 19:56:32.742278 10515 consensus_queue.cc:237] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:32.743952 8839 raft_consensus.cc:1261] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 19:56:32.744417 11897 consensus_queue.cc:1035] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 19:56:32.746789 11524 raft_consensus.cc:2932] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
I20240501 19:56:32.749372 8695 catalog_manager.cc:5484] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a reported cstate change: config changed from index -1 to 3, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New cstate: current_term: 2 leader_uuid: "14798909ff724444a8be03c45c41929a" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:32.750133 8839 raft_consensus.cc:2932] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
W20240501 19:56:32.751809 10449 consensus_peers.cc:485] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a -> Peer 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72:33401): Couldn't send request to peer 4663c01c9815427ab4bba05e1d1e44aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c6e64c2bfdce4caf9c91618400c7f5d6. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:32.758054 8967 consensus_queue.cc:237] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 246, Committed index: 246, Last appended: 1.246, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:32.759460 9101 raft_consensus.cc:1261] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 246. Preceding OpId from leader: term: 1 index: 247. (index mismatch)
I20240501 19:56:32.759841 11907 consensus_queue.cc:1035] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 247, Last known committed idx: 246, Time since last communication: 0.000s
I20240501 19:56:32.761552 11138 raft_consensus.cc:2932] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.247: config changed from index -1 to 247, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
I20240501 19:56:32.764276 9101 raft_consensus.cc:2932] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Committing config change with OpId 1.247: config changed from index -1 to 247, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New config: { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } } }
W20240501 19:56:32.764637 8905 consensus_peers.cc:485] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b -> Peer 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72:33401): Couldn't send request to peer 4663c01c9815427ab4bba05e1d1e44aa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6db903f127ce471d9cb87537a3cb7286. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:32.765893 8695 catalog_manager.cc:5484] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index -1 to 247, NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:33.160197 11950 ts_tablet_manager.cc:902] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: Initiating tablet copy from peer ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725)
I20240501 19:56:33.160899 11950 tablet_copy_client.cc:308] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:43725
I20240501 19:56:33.164542 8990 tablet_copy_service.cc:140] P ac13bdb319c448dba072fb9e62d6893b: Received BeginTabletCopySession request for tablet 6db903f127ce471d9cb87537a3cb7286 from peer 4663c01c9815427ab4bba05e1d1e44aa ({username='slave'} at 127.5.139.72:40071)
I20240501 19:56:33.164642 8990 tablet_copy_service.cc:161] P ac13bdb319c448dba072fb9e62d6893b: Beginning new tablet copy session on tablet 6db903f127ce471d9cb87537a3cb7286 from peer 4663c01c9815427ab4bba05e1d1e44aa at {username='slave'} at 127.5.139.72:40071: session id = 4663c01c9815427ab4bba05e1d1e44aa-6db903f127ce471d9cb87537a3cb7286
I20240501 19:56:33.165393 8990 tablet_copy_source_session.cc:205] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:33.168269 11950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6db903f127ce471d9cb87537a3cb7286. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.172832 11950 tablet_copy_client.cc:783] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:33.173111 11950 tablet_copy_client.cc:647] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:33.175011 11950 tablet_copy_client.cc:523] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:33.176813 11950 tablet_bootstrap.cc:492] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap starting.
I20240501 19:56:33.186179 11956 ts_tablet_manager.cc:902] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: Initiating tablet copy from peer b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389)
I20240501 19:56:33.186697 11956 tablet_copy_client.cc:308] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:35389
I20240501 19:56:33.190315 9121 tablet_copy_service.cc:140] P b3b45c9058d84b4e9ace42c9154ab0e0: Received BeginTabletCopySession request for tablet aa477b2bd34d4b74b245314148e4851d from peer 4663c01c9815427ab4bba05e1d1e44aa ({username='slave'} at 127.5.139.72:43053)
I20240501 19:56:33.190418 9121 tablet_copy_service.cc:161] P b3b45c9058d84b4e9ace42c9154ab0e0: Beginning new tablet copy session on tablet aa477b2bd34d4b74b245314148e4851d from peer 4663c01c9815427ab4bba05e1d1e44aa at {username='slave'} at 127.5.139.72:43053: session id = 4663c01c9815427ab4bba05e1d1e44aa-aa477b2bd34d4b74b245314148e4851d
I20240501 19:56:33.191237 9121 tablet_copy_source_session.cc:205] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:33.191992 11956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa477b2bd34d4b74b245314148e4851d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.193580 11956 tablet_copy_client.cc:783] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:33.193710 11956 tablet_copy_client.cc:647] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:33.195351 11956 tablet_copy_client.cc:523] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:33.196242 11956 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap starting.
I20240501 19:56:33.197741 11950 log.cc:824] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:33.276798 11778 heartbeater.cc:503] Master 127.5.139.126:34427 was elected leader, sending a full tablet report...
I20240501 19:56:33.285207 11983 ts_tablet_manager.cc:902] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: Initiating tablet copy from peer 14798909ff724444a8be03c45c41929a (127.5.139.71:38393)
I20240501 19:56:33.285815 11983 tablet_copy_client.cc:308] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.71:38393
I20240501 19:56:33.290161 10535 tablet_copy_service.cc:140] P 14798909ff724444a8be03c45c41929a: Received BeginTabletCopySession request for tablet c6e64c2bfdce4caf9c91618400c7f5d6 from peer 4663c01c9815427ab4bba05e1d1e44aa ({username='slave'} at 127.5.139.72:36381)
I20240501 19:56:33.290302 10535 tablet_copy_service.cc:161] P 14798909ff724444a8be03c45c41929a: Beginning new tablet copy session on tablet c6e64c2bfdce4caf9c91618400c7f5d6 from peer 4663c01c9815427ab4bba05e1d1e44aa at {username='slave'} at 127.5.139.72:36381: session id = 4663c01c9815427ab4bba05e1d1e44aa-c6e64c2bfdce4caf9c91618400c7f5d6
I20240501 19:56:33.291754 10535 tablet_copy_source_session.cc:205] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:33.292805 11983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6e64c2bfdce4caf9c91618400c7f5d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.294982 11983 tablet_copy_client.cc:783] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:33.295166 11983 tablet_copy_client.cc:647] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:33.296569 11983 tablet_copy_client.cc:523] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:33.297709 11983 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap starting.
I20240501 19:56:33.301503 11983 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:33.301856 11983 tablet_bootstrap.cc:492] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap complete.
I20240501 19:56:33.302192 11983 ts_tablet_manager.cc:1367] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:33.305477 11983 raft_consensus.cc:348] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:33.305811 11983 raft_consensus.cc:729] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4663c01c9815427ab4bba05e1d1e44aa, State: Initialized, Role: LEARNER
I20240501 19:56:33.306236 11983 consensus_queue.cc:260] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:33.307309 11983 ts_tablet_manager.cc:1398] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:33.308105 10535 tablet_copy_service.cc:342] P 14798909ff724444a8be03c45c41929a: Request end of tablet copy session 4663c01c9815427ab4bba05e1d1e44aa-c6e64c2bfdce4caf9c91618400c7f5d6 received from {username='slave'} at 127.5.139.72:36381
I20240501 19:56:33.308203 10535 tablet_copy_service.cc:433] P 14798909ff724444a8be03c45c41929a: ending tablet copy session 4663c01c9815427ab4bba05e1d1e44aa-c6e64c2bfdce4caf9c91618400c7f5d6 on tablet c6e64c2bfdce4caf9c91618400c7f5d6 with peer 4663c01c9815427ab4bba05e1d1e44aa
I20240501 19:56:33.341198 11983 ts_tablet_manager.cc:902] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: Initiating tablet copy from peer 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483)
I20240501 19:56:33.341717 11983 tablet_copy_client.cc:308] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.69:42483
I20240501 19:56:33.346154 10272 tablet_copy_service.cc:140] P 6977f9a52e9b4d06bcb12f9bee61553b: Received BeginTabletCopySession request for tablet d827d0b0d5ba4969908768c96f92c1f3 from peer 4663c01c9815427ab4bba05e1d1e44aa ({username='slave'} at 127.5.139.72:34307)
I20240501 19:56:33.346313 10272 tablet_copy_service.cc:161] P 6977f9a52e9b4d06bcb12f9bee61553b: Beginning new tablet copy session on tablet d827d0b0d5ba4969908768c96f92c1f3 from peer 4663c01c9815427ab4bba05e1d1e44aa at {username='slave'} at 127.5.139.72:34307: session id = 4663c01c9815427ab4bba05e1d1e44aa-d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:33.347862 10272 tablet_copy_source_session.cc:205] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:33.349102 11983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d827d0b0d5ba4969908768c96f92c1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:33.351105 11983 tablet_copy_client.cc:783] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:33.351238 11983 tablet_copy_client.cc:647] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:33.353461 11950 tablet_bootstrap.cc:492] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=247 overwritten=0 applied=247 ignored=0} inserts{seen=932 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:33.353855 11950 tablet_bootstrap.cc:492] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap complete.
I20240501 19:56:33.354077 11950 ts_tablet_manager.cc:1367] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: Time spent bootstrapping tablet: real 0.177s user 0.169s sys 0.009s
I20240501 19:56:33.354280 11983 tablet_copy_client.cc:523] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:33.354725 11950 raft_consensus.cc:348] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:33.354825 11950 raft_consensus.cc:729] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4663c01c9815427ab4bba05e1d1e44aa, State: Initialized, Role: LEARNER
I20240501 19:56:33.354909 11950 consensus_queue.cc:260] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 247, Last appended: 1.247, Last appended by leader: 247, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 247 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:33.355551 11950 ts_tablet_manager.cc:1398] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:33.356173 8990 tablet_copy_service.cc:342] P ac13bdb319c448dba072fb9e62d6893b: Request end of tablet copy session 4663c01c9815427ab4bba05e1d1e44aa-6db903f127ce471d9cb87537a3cb7286 received from {username='slave'} at 127.5.139.72:40071
I20240501 19:56:33.356254 8990 tablet_copy_service.cc:433] P ac13bdb319c448dba072fb9e62d6893b: ending tablet copy session 4663c01c9815427ab4bba05e1d1e44aa-6db903f127ce471d9cb87537a3cb7286 on tablet 6db903f127ce471d9cb87537a3cb7286 with peer 4663c01c9815427ab4bba05e1d1e44aa
I20240501 19:56:33.357215 11983 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap starting.
I20240501 19:56:33.382133 11956 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=280 overwritten=0 applied=280 ignored=0} inserts{seen=908 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:33.382484 11956 tablet_bootstrap.cc:492] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap complete.
I20240501 19:56:33.382695 11956 ts_tablet_manager.cc:1367] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: Time spent bootstrapping tablet: real 0.186s user 0.178s sys 0.004s
I20240501 19:56:33.383174 11956 raft_consensus.cc:348] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:33.383309 11956 raft_consensus.cc:729] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4663c01c9815427ab4bba05e1d1e44aa, State: Initialized, Role: LEARNER
I20240501 19:56:33.383442 11956 consensus_queue.cc:260] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 280, Last appended: 1.280, Last appended by leader: 280, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 280 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:33.383736 11956 ts_tablet_manager.cc:1398] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:33.384338 9121 tablet_copy_service.cc:342] P b3b45c9058d84b4e9ace42c9154ab0e0: Request end of tablet copy session 4663c01c9815427ab4bba05e1d1e44aa-aa477b2bd34d4b74b245314148e4851d received from {username='slave'} at 127.5.139.72:43053
I20240501 19:56:33.384430 9121 tablet_copy_service.cc:433] P b3b45c9058d84b4e9ace42c9154ab0e0: ending tablet copy session 4663c01c9815427ab4bba05e1d1e44aa-aa477b2bd34d4b74b245314148e4851d on tablet aa477b2bd34d4b74b245314148e4851d with peer 4663c01c9815427ab4bba05e1d1e44aa
I20240501 19:56:33.528308 11983 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=275 overwritten=0 applied=275 ignored=0} inserts{seen=940 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:33.528643 11983 tablet_bootstrap.cc:492] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: Bootstrap complete.
I20240501 19:56:33.528832 11983 ts_tablet_manager.cc:1367] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: Time spent bootstrapping tablet: real 0.172s user 0.168s sys 0.000s
I20240501 19:56:33.529256 11983 raft_consensus.cc:348] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:33.529362 11983 raft_consensus.cc:729] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4663c01c9815427ab4bba05e1d1e44aa, State: Initialized, Role: LEARNER
I20240501 19:56:33.529449 11983 consensus_queue.cc:260] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 275, Last appended: 2.275, Last appended by leader: 275, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 275 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: true } }
I20240501 19:56:33.529695 11983 ts_tablet_manager.cc:1398] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:33.530267 10272 tablet_copy_service.cc:342] P 6977f9a52e9b4d06bcb12f9bee61553b: Request end of tablet copy session 4663c01c9815427ab4bba05e1d1e44aa-d827d0b0d5ba4969908768c96f92c1f3 received from {username='slave'} at 127.5.139.72:34307
I20240501 19:56:33.530354 10272 tablet_copy_service.cc:433] P 6977f9a52e9b4d06bcb12f9bee61553b: ending tablet copy session 4663c01c9815427ab4bba05e1d1e44aa-d827d0b0d5ba4969908768c96f92c1f3 on tablet d827d0b0d5ba4969908768c96f92c1f3 with peer 4663c01c9815427ab4bba05e1d1e44aa
I20240501 19:56:33.682508 11732 raft_consensus.cc:1203] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.246->[1.247-1.247] Dedup: 1.247->[]
I20240501 19:56:33.737087 11732 raft_consensus.cc:1203] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240501 19:56:33.783825 11732 raft_consensus.cc:1203] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.279->[1.280-1.280] Dedup: 1.280->[]
I20240501 19:56:33.827759 11732 raft_consensus.cc:1203] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEARNER]: Deduplicated request from leader. Original: 2.274->[2.275-2.275] Dedup: 2.275->[]
I20240501 19:56:34.138693 12039 raft_consensus.cc:1050] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a: attempting to promote NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa to VOTER
I20240501 19:56:34.139147 12039 consensus_queue.cc:237] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.140560 8839 raft_consensus.cc:1261] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:34.140584 11732 raft_consensus.cc:1261] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEARNER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 19:56:34.140906 12039 consensus_queue.cc:1035] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.141189 11900 consensus_queue.cc:1035] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:34.143153 12039 raft_consensus.cc:2932] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.143700 8839 raft_consensus.cc:2932] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.144490 11732 raft_consensus.cc:2932] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.146257 8695 catalog_manager.cc:5484] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a reported cstate change: config changed from index 3 to 4, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "14798909ff724444a8be03c45c41929a" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "0f2b49c6d3dc49fabee40802e73ac982" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38731 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:34.150025 10515 consensus_queue.cc:237] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.150995 11732 raft_consensus.cc:1261] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Refusing update from remote peer 14798909ff724444a8be03c45c41929a: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240501 19:56:34.151376 12039 consensus_queue.cc:1035] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [LEADER]: Connected to new peer: Peer: permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:34.152973 12038 raft_consensus.cc:2932] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.153424 11732 raft_consensus.cc:2932] T c6e64c2bfdce4caf9c91618400c7f5d6 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.154563 8682 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c6e64c2bfdce4caf9c91618400c7f5d6 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:34.155414 8694 catalog_manager.cc:5484] T c6e64c2bfdce4caf9c91618400c7f5d6 P 14798909ff724444a8be03c45c41929a reported cstate change: config changed from index 4 to 5, VOTER 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65) evicted. New cstate: current_term: 2 leader_uuid: "14798909ff724444a8be03c45c41929a" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "14798909ff724444a8be03c45c41929a" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 38393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:34.157508 8819 tablet_service.cc:1514] Processing DeleteTablet for tablet c6e64c2bfdce4caf9c91618400c7f5d6 with delete_type TABLET_DATA_TOMBSTONED (TS 0f2b49c6d3dc49fabee40802e73ac982 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:48934
I20240501 19:56:34.157872 12092 tablet_replica.cc:330] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: stopping tablet replica
I20240501 19:56:34.158026 12092 raft_consensus.cc:2229] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:34.158160 12092 raft_consensus.cc:2258] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:34.159431 12092 ts_tablet_manager.cc:1875] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:34.161231 12092 ts_tablet_manager.cc:1888] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240501 19:56:34.161301 12092 log.cc:1196] T c6e64c2bfdce4caf9c91618400c7f5d6 P 0f2b49c6d3dc49fabee40802e73ac982: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/c6e64c2bfdce4caf9c91618400c7f5d6
I20240501 19:56:34.161782 8681 catalog_manager.cc:4830] TS 0f2b49c6d3dc49fabee40802e73ac982 (127.5.139.65:38731): tablet c6e64c2bfdce4caf9c91618400c7f5d6 (table TestTable [id=667da58c9fb24a419759f30314644e23]) successfully deleted
I20240501 19:56:34.246564 12094 raft_consensus.cc:1050] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: attempting to promote NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa to VOTER
I20240501 19:56:34.246982 12094 consensus_queue.cc:237] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 247, Committed index: 247, Last appended: 1.247, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.248267 11732 raft_consensus.cc:1261] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 1 LEARNER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240501 19:56:34.248412 9101 raft_consensus.cc:1261] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Refusing update from remote peer ac13bdb319c448dba072fb9e62d6893b: Log matching property violated. Preceding OpId in replica: term: 1 index: 247. Preceding OpId from leader: term: 1 index: 248. (index mismatch)
I20240501 19:56:34.248764 12096 consensus_queue.cc:1035] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240501 19:56:34.248958 12094 consensus_queue.cc:1035] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 248, Last known committed idx: 247, Time since last communication: 0.000s
I20240501 19:56:34.250715 12096 raft_consensus.cc:2932] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Committing config change with OpId 1.248: config changed from index 247 to 248, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.251138 11732 raft_consensus.cc:2932] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 247 to 248, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.252658 9101 raft_consensus.cc:2932] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Committing config change with OpId 1.248: config changed from index 247 to 248, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.253302 8688 catalog_manager.cc:5484] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b reported cstate change: config changed from index 247 to 248, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ac13bdb319c448dba072fb9e62d6893b" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:34.310823 12099 raft_consensus.cc:1050] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: attempting to promote NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa to VOTER
I20240501 19:56:34.311223 12099 consensus_queue.cc:237] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 280, Committed index: 280, Last appended: 1.280, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.312541 11732 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 1 LEARNER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240501 19:56:34.312541 10252 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 280. Preceding OpId from leader: term: 1 index: 281. (index mismatch)
I20240501 19:56:34.312975 12101 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240501 19:56:34.313218 12099 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 281, Last known committed idx: 280, Time since last communication: 0.000s
I20240501 19:56:34.314952 11877 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Committing config change with OpId 1.281: config changed from index 280 to 281, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.315464 11732 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.317019 10252 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Committing config change with OpId 1.281: config changed from index 280 to 281, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.317613 8695 catalog_manager.cc:5484] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 280 to 281, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:34.329964 12043 raft_consensus.cc:1050] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: attempting to promote NON_VOTER 4663c01c9815427ab4bba05e1d1e44aa to VOTER
I20240501 19:56:34.330340 12043 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 275, Committed index: 275, Last appended: 2.275, Last appended by leader: 150, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.331403 11732 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEARNER]: Refusing update from remote peer 6977f9a52e9b4d06bcb12f9bee61553b: Log matching property violated. Preceding OpId in replica: term: 2 index: 275. Preceding OpId from leader: term: 2 index: 276. (index mismatch)
I20240501 19:56:34.331591 10381 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Refusing update from remote peer 6977f9a52e9b4d06bcb12f9bee61553b: Log matching property violated. Preceding OpId in replica: term: 2 index: 275. Preceding OpId from leader: term: 2 index: 276. (index mismatch)
I20240501 19:56:34.331782 12104 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I20240501 19:56:34.331988 12106 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 276, Last known committed idx: 275, Time since last communication: 0.000s
I20240501 19:56:34.333751 12043 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Committing config change with OpId 2.276: config changed from index 275 to 276, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.334164 11732 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Committing config change with OpId 2.276: config changed from index 275 to 276, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.334288 10381 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Committing config change with OpId 2.276: config changed from index 275 to 276, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:34.336607 8694 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b reported cstate change: config changed from index 275 to 276, 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:34.367874 8967 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6db903f127ce471d9cb87537a3cb7286"
dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57874
I20240501 19:56:34.368108 8967 raft_consensus.cc:595] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:34.378082 9101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa477b2bd34d4b74b245314148e4851d"
dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49234
I20240501 19:56:34.378211 9101 raft_consensus.cc:595] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:34.387662 10252 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3"
dest_uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50070
I20240501 19:56:34.387816 10252 raft_consensus.cc:595] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Received request to transfer leadership
I20240501 19:56:34.397456 8967 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6db903f127ce471d9cb87537a3cb7286"
dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57886
I20240501 19:56:34.397567 8967 raft_consensus.cc:595] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:34.407299 9101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa477b2bd34d4b74b245314148e4851d"
dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49252
I20240501 19:56:34.407410 9101 raft_consensus.cc:595] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:34.416635 10252 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3"
dest_uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50090
I20240501 19:56:34.416757 10252 raft_consensus.cc:595] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Received request to transfer leadership
I20240501 19:56:34.627619 8967 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6db903f127ce471d9cb87537a3cb7286"
dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57896
I20240501 19:56:34.627768 8967 raft_consensus.cc:595] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:34.637387 9101 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aa477b2bd34d4b74b245314148e4851d"
dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49268
I20240501 19:56:34.637504 9101 raft_consensus.cc:595] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:34.647172 10252 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3"
dest_uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:50108
I20240501 19:56:34.647294 10252 raft_consensus.cc:595] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Received request to transfer leadership
I20240501 19:56:34.688334 12022 raft_consensus.cc:979] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: : Instructing follower b3b45c9058d84b4e9ace42c9154ab0e0 to start an election
I20240501 19:56:34.688463 12022 raft_consensus.cc:1067] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Signalling peer b3b45c9058d84b4e9ace42c9154ab0e0 to start an election
I20240501 19:56:34.688930 9101 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6db903f127ce471d9cb87537a3cb7286"
dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
from {username='slave'} at 127.5.139.66:40965
I20240501 19:56:34.689088 9101 raft_consensus.cc:482] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:34.689143 9101 raft_consensus.cc:3037] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.689922 9101 raft_consensus.cc:504] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.690243 9101 leader_election.cc:288] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 2 election: Requested vote from peers ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725), 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72:33401)
I20240501 19:56:34.690573 8967 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db903f127ce471d9cb87537a3cb7286" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 248 } } ignore_live_leader: true dest_uuid: "ac13bdb319c448dba072fb9e62d6893b"
I20240501 19:56:34.690717 8967 raft_consensus.cc:3032] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:34.690691 11732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6db903f127ce471d9cb87537a3cb7286" candidate_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" candidate_term: 2 candidate_status { last_received { term: 1 index: 248 } } ignore_live_leader: true dest_uuid: "4663c01c9815427ab4bba05e1d1e44aa"
I20240501 19:56:34.690764 8967 raft_consensus.cc:729] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ac13bdb319c448dba072fb9e62d6893b, State: Running, Role: LEADER
I20240501 19:56:34.690856 11732 raft_consensus.cc:3037] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.690912 8967 consensus_queue.cc:260] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 1.248, Last appended by leader: 248, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.691110 8967 raft_consensus.cc:3037] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.691694 11732 raft_consensus.cc:2454] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 2.
I20240501 19:56:34.691860 8967 raft_consensus.cc:2454] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3b45c9058d84b4e9ace42c9154ab0e0 in term 2.
I20240501 19:56:34.692045 9036 leader_election.cc:302] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4663c01c9815427ab4bba05e1d1e44aa, b3b45c9058d84b4e9ace42c9154ab0e0; no voters:
I20240501 19:56:34.692255 11877 raft_consensus.cc:2781] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:34.692337 11877 raft_consensus.cc:686] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 LEADER]: Becoming Leader. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:34.692485 11877 consensus_queue.cc:237] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 248, Committed index: 248, Last appended: 1.248, Last appended by leader: 248, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.694020 8688 catalog_manager.cc:5484] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: term changed from 1 to 2, leader changed from ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) to b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67). New cstate: current_term: 2 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 248 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:34.766368 12104 raft_consensus.cc:979] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: : Instructing follower 4663c01c9815427ab4bba05e1d1e44aa to start an election
I20240501 19:56:34.766485 12104 raft_consensus.cc:1067] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Signalling peer 4663c01c9815427ab4bba05e1d1e44aa to start an election
I20240501 19:56:34.767123 11732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3"
dest_uuid: "4663c01c9815427ab4bba05e1d1e44aa"
from {username='slave'} at 127.5.139.69:54217
I20240501 19:56:34.767318 11732 raft_consensus.cc:482] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:34.767392 11732 raft_consensus.cc:3037] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:34.768333 11732 raft_consensus.cc:504] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.769276 11732 leader_election.cc:288] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [CANDIDATE]: Term 3 election: Requested vote from peers 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483), 6b4f42453a864314adc86aba2c3b3556 (127.5.139.70:43441)
I20240501 19:56:34.769630 10252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3" candidate_uuid: "4663c01c9815427ab4bba05e1d1e44aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 276 } } ignore_live_leader: true dest_uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
I20240501 19:56:34.769762 10252 raft_consensus.cc:3032] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Stepping down as leader of term 2
I20240501 19:56:34.769811 10252 raft_consensus.cc:729] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6977f9a52e9b4d06bcb12f9bee61553b, State: Running, Role: LEADER
I20240501 19:56:34.769982 10252 consensus_queue.cc:260] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 2.276, Last appended by leader: 276, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.770186 10252 raft_consensus.cc:3037] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:34.771173 10252 raft_consensus.cc:2454] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4663c01c9815427ab4bba05e1d1e44aa in term 3.
I20240501 19:56:34.771519 11665 leader_election.cc:302] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4663c01c9815427ab4bba05e1d1e44aa, 6977f9a52e9b4d06bcb12f9bee61553b; no voters:
I20240501 19:56:34.771747 11989 raft_consensus.cc:2781] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:56:34.771973 11989 raft_consensus.cc:686] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 3 LEADER]: Becoming Leader. State: Replica: 4663c01c9815427ab4bba05e1d1e44aa, State: Running, Role: LEADER
I20240501 19:56:34.772271 11989 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 276, Committed index: 276, Last appended: 2.276, Last appended by leader: 276, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.773425 10381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d827d0b0d5ba4969908768c96f92c1f3" candidate_uuid: "4663c01c9815427ab4bba05e1d1e44aa" candidate_term: 3 candidate_status { last_received { term: 2 index: 276 } } ignore_live_leader: true dest_uuid: "6b4f42453a864314adc86aba2c3b3556"
I20240501 19:56:34.773550 10381 raft_consensus.cc:3037] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:34.774535 10381 raft_consensus.cc:2454] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4663c01c9815427ab4bba05e1d1e44aa in term 3.
I20240501 19:56:34.774894 8688 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa reported cstate change: term changed from 2 to 3, leader changed from 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) to 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72). New cstate: current_term: 3 leader_uuid: "4663c01c9815427ab4bba05e1d1e44aa" committed_config { opid_index: 276 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:34.814268 12099 raft_consensus.cc:979] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: : Instructing follower 4663c01c9815427ab4bba05e1d1e44aa to start an election
I20240501 19:56:34.814375 12099 raft_consensus.cc:1067] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Signalling peer 4663c01c9815427ab4bba05e1d1e44aa to start an election
I20240501 19:56:34.814805 11732 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aa477b2bd34d4b74b245314148e4851d"
dest_uuid: "4663c01c9815427ab4bba05e1d1e44aa"
from {username='slave'} at 127.5.139.67:47675
I20240501 19:56:34.814934 11732 raft_consensus.cc:482] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:34.814983 11732 raft_consensus.cc:3037] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.815747 11732 raft_consensus.cc:504] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.816154 11732 leader_election.cc:288] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [CANDIDATE]: Term 2 election: Requested vote from peers b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389), 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483)
I20240501 19:56:34.816484 9101 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa477b2bd34d4b74b245314148e4851d" candidate_uuid: "4663c01c9815427ab4bba05e1d1e44aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 281 } } ignore_live_leader: true dest_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0"
I20240501 19:56:34.816613 9101 raft_consensus.cc:3032] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:34.816606 10252 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa477b2bd34d4b74b245314148e4851d" candidate_uuid: "4663c01c9815427ab4bba05e1d1e44aa" candidate_term: 2 candidate_status { last_received { term: 1 index: 281 } } ignore_live_leader: true dest_uuid: "6977f9a52e9b4d06bcb12f9bee61553b"
I20240501 19:56:34.816664 9101 raft_consensus.cc:729] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b3b45c9058d84b4e9ace42c9154ab0e0, State: Running, Role: LEADER
I20240501 19:56:34.816735 10252 raft_consensus.cc:3037] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.816821 9101 consensus_queue.cc:260] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 281, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.817019 9101 raft_consensus.cc:3037] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:34.817814 10252 raft_consensus.cc:2454] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4663c01c9815427ab4bba05e1d1e44aa in term 2.
I20240501 19:56:34.818060 9101 raft_consensus.cc:2454] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4663c01c9815427ab4bba05e1d1e44aa in term 2.
I20240501 19:56:34.818212 11665 leader_election.cc:302] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4663c01c9815427ab4bba05e1d1e44aa, 6977f9a52e9b4d06bcb12f9bee61553b; no voters:
I20240501 19:56:34.818521 11989 raft_consensus.cc:2781] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:34.818603 11989 raft_consensus.cc:686] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEADER]: Becoming Leader. State: Replica: 4663c01c9815427ab4bba05e1d1e44aa, State: Running, Role: LEADER
I20240501 19:56:34.818737 11989 consensus_queue.cc:237] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 281, Committed index: 281, Last appended: 1.281, Last appended by leader: 281, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:34.820544 8688 catalog_manager.cc:5484] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa reported cstate change: term changed from 1 to 2, leader changed from b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) to 4663c01c9815427ab4bba05e1d1e44aa (127.5.139.72). New cstate: current_term: 2 leader_uuid: "4663c01c9815427ab4bba05e1d1e44aa" committed_config { opid_index: 281 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.163122 10252 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 3 FOLLOWER]: Refusing update from remote peer 4663c01c9815427ab4bba05e1d1e44aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 276. Preceding OpId from leader: term: 3 index: 277. (index mismatch)
I20240501 19:56:35.163769 12207 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.000s
I20240501 19:56:35.166730 10381 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 3 FOLLOWER]: Refusing update from remote peer 4663c01c9815427ab4bba05e1d1e44aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 276. Preceding OpId from leader: term: 3 index: 277. (index mismatch)
I20240501 19:56:35.167292 12207 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 277, Last known committed idx: 276, Time since last communication: 0.000s
I20240501 19:56:35.261303 9101 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Refusing update from remote peer 4663c01c9815427ab4bba05e1d1e44aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20240501 19:56:35.261701 12207 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240501 19:56:35.264199 10252 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Refusing update from remote peer 4663c01c9815427ab4bba05e1d1e44aa: Log matching property violated. Preceding OpId in replica: term: 1 index: 281. Preceding OpId from leader: term: 2 index: 282. (index mismatch)
I20240501 19:56:35.265225 11989 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 282, Last known committed idx: 281, Time since last communication: 0.000s
I20240501 19:56:35.314177 11732 raft_consensus.cc:1261] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 2 index: 249. (index mismatch)
I20240501 19:56:35.314729 12099 consensus_queue.cc:1035] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240501 19:56:35.316397 8967 raft_consensus.cc:1261] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 2 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 1 index: 248. Preceding OpId from leader: term: 2 index: 249. (index mismatch)
I20240501 19:56:35.316895 12260 consensus_queue.cc:1035] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac13bdb319c448dba072fb9e62d6893b" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 43725 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 249, Last known committed idx: 248, Time since last communication: 0.000s
I20240501 19:56:35.318879 9101 consensus_queue.cc:237] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 249, Committed index: 249, Last appended: 2.249, Last appended by leader: 248, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:35.319969 11732 raft_consensus.cc:1261] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Refusing update from remote peer b3b45c9058d84b4e9ace42c9154ab0e0: Log matching property violated. Preceding OpId in replica: term: 2 index: 249. Preceding OpId from leader: term: 2 index: 250. (index mismatch)
I20240501 19:56:35.320279 12260 consensus_queue.cc:1035] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 250, Last known committed idx: 249, Time since last communication: 0.000s
I20240501 19:56:35.321744 12099 raft_consensus.cc:2932] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 LEADER]: Committing config change with OpId 2.250: config changed from index 248 to 250, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:35.322228 11732 raft_consensus.cc:2932] T 6db903f127ce471d9cb87537a3cb7286 P 4663c01c9815427ab4bba05e1d1e44aa [term 2 FOLLOWER]: Committing config change with OpId 2.250: config changed from index 248 to 250, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New config: { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:35.323401 8684 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6db903f127ce471d9cb87537a3cb7286 with cas_config_opid_index 248: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.330163 8695 catalog_manager.cc:5484] T 6db903f127ce471d9cb87537a3cb7286 P b3b45c9058d84b4e9ace42c9154ab0e0 reported cstate change: config changed from index 248 to 250, VOTER ac13bdb319c448dba072fb9e62d6893b (127.5.139.66) evicted. New cstate: current_term: 2 leader_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" committed_config { opid_index: 250 OBSOLETE_local: false peers { permanent_uuid: "b3b45c9058d84b4e9ace42c9154ab0e0" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 35389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.332790 8949 tablet_service.cc:1514] Processing DeleteTablet for tablet 6db903f127ce471d9cb87537a3cb7286 with delete_type TABLET_DATA_TOMBSTONED (TS ac13bdb319c448dba072fb9e62d6893b not found in new config with opid_index 250) from {username='slave'} at 127.0.0.1:45626
I20240501 19:56:35.333151 12266 tablet_replica.cc:330] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: stopping tablet replica
I20240501 19:56:35.333281 12266 raft_consensus.cc:2229] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.333405 12266 raft_consensus.cc:2258] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.334692 12266 ts_tablet_manager.cc:1875] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.336968 12266 ts_tablet_manager.cc:1888] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.249
I20240501 19:56:35.337072 12266 log.cc:1196] T 6db903f127ce471d9cb87537a3cb7286 P ac13bdb319c448dba072fb9e62d6893b: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/6db903f127ce471d9cb87537a3cb7286
I20240501 19:56:35.337606 8682 catalog_manager.cc:4830] TS ac13bdb319c448dba072fb9e62d6893b (127.5.139.66:43725): tablet 6db903f127ce471d9cb87537a3cb7286 (table rebalance_test_table_0 [id=4bd9cb5cb29f48658b0e22bd5e3743bf]) successfully deleted
I20240501 19:56:35.560081 11732 consensus_queue.cc:237] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 277, Committed index: 277, Last appended: 3.277, Last appended by leader: 276, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:35.561094 10381 raft_consensus.cc:1261] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 3 FOLLOWER]: Refusing update from remote peer 4663c01c9815427ab4bba05e1d1e44aa: Log matching property violated. Preceding OpId in replica: term: 3 index: 277. Preceding OpId from leader: term: 3 index: 278. (index mismatch)
I20240501 19:56:35.561475 12207 consensus_queue.cc:1035] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 278, Last known committed idx: 277, Time since last communication: 0.000s
I20240501 19:56:35.562928 12292 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa [term 3 LEADER]: Committing config change with OpId 3.278: config changed from index 276 to 278, VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:35.563258 10381 raft_consensus.cc:2932] T d827d0b0d5ba4969908768c96f92c1f3 P 6b4f42453a864314adc86aba2c3b3556 [term 3 FOLLOWER]: Committing config change with OpId 3.278: config changed from index 276 to 278, VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) evicted. New config: { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:35.564364 8683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d827d0b0d5ba4969908768c96f92c1f3 with cas_config_opid_index 276: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.565196 8688 catalog_manager.cc:5484] T d827d0b0d5ba4969908768c96f92c1f3 P 4663c01c9815427ab4bba05e1d1e44aa reported cstate change: config changed from index 276 to 278, VOTER 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69) evicted. New cstate: current_term: 3 leader_uuid: "4663c01c9815427ab4bba05e1d1e44aa" committed_config { opid_index: 278 OBSOLETE_local: false peers { permanent_uuid: "6b4f42453a864314adc86aba2c3b3556" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 43441 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.567524 10232 tablet_service.cc:1514] Processing DeleteTablet for tablet d827d0b0d5ba4969908768c96f92c1f3 with delete_type TABLET_DATA_TOMBSTONED (TS 6977f9a52e9b4d06bcb12f9bee61553b not found in new config with opid_index 278) from {username='slave'} at 127.0.0.1:49888
I20240501 19:56:35.567970 12294 tablet_replica.cc:330] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: stopping tablet replica
I20240501 19:56:35.568233 12294 raft_consensus.cc:2229] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.568368 12294 raft_consensus.cc:2258] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.569880 12294 ts_tablet_manager.cc:1875] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.571691 12294 ts_tablet_manager.cc:1888] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.277
I20240501 19:56:35.571764 12294 log.cc:1196] T d827d0b0d5ba4969908768c96f92c1f3 P 6977f9a52e9b4d06bcb12f9bee61553b: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/wals/d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:35.572247 8681 catalog_manager.cc:4830] TS 6977f9a52e9b4d06bcb12f9bee61553b (127.5.139.69:42483): tablet d827d0b0d5ba4969908768c96f92c1f3 (table rebalance_test_table_4 [id=71ec2b5fefea4dca96f79662a350a8f3]) successfully deleted
I20240501 19:56:35.707312 11732 consensus_queue.cc:237] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 282, Committed index: 282, Last appended: 2.282, Last appended by leader: 281, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } }
I20240501 19:56:35.708292 10252 raft_consensus.cc:1261] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Refusing update from remote peer 4663c01c9815427ab4bba05e1d1e44aa: Log matching property violated. Preceding OpId in replica: term: 2 index: 282. Preceding OpId from leader: term: 2 index: 283. (index mismatch)
I20240501 19:56:35.708667 12207 consensus_queue.cc:1035] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 283, Last known committed idx: 282, Time since last communication: 0.000s
I20240501 19:56:35.710093 12292 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa [term 2 LEADER]: Committing config change with OpId 2.283: config changed from index 281 to 283, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:35.710489 10252 raft_consensus.cc:2932] T aa477b2bd34d4b74b245314148e4851d P 6977f9a52e9b4d06bcb12f9bee61553b [term 2 FOLLOWER]: Committing config change with OpId 2.283: config changed from index 281 to 283, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New config: { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } } }
I20240501 19:56:35.711723 8683 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet aa477b2bd34d4b74b245314148e4851d with cas_config_opid_index 281: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:35.712603 8695 catalog_manager.cc:5484] T aa477b2bd34d4b74b245314148e4851d P 4663c01c9815427ab4bba05e1d1e44aa reported cstate change: config changed from index 281 to 283, VOTER b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67) evicted. New cstate: current_term: 2 leader_uuid: "4663c01c9815427ab4bba05e1d1e44aa" committed_config { opid_index: 283 OBSOLETE_local: false peers { permanent_uuid: "6977f9a52e9b4d06bcb12f9bee61553b" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 42483 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4663c01c9815427ab4bba05e1d1e44aa" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 33401 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:35.715150 9080 tablet_service.cc:1514] Processing DeleteTablet for tablet aa477b2bd34d4b74b245314148e4851d with delete_type TABLET_DATA_TOMBSTONED (TS b3b45c9058d84b4e9ace42c9154ab0e0 not found in new config with opid_index 283) from {username='slave'} at 127.0.0.1:44930
I20240501 19:56:35.715538 12296 tablet_replica.cc:330] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: stopping tablet replica
I20240501 19:56:35.715698 12296 raft_consensus.cc:2229] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:35.715849 12296 raft_consensus.cc:2258] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:35.717375 12296 ts_tablet_manager.cc:1875] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:35.719362 12296 ts_tablet_manager.cc:1888] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.282
I20240501 19:56:35.719434 12296 log.cc:1196] T aa477b2bd34d4b74b245314148e4851d P b3b45c9058d84b4e9ace42c9154ab0e0: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/aa477b2bd34d4b74b245314148e4851d
I20240501 19:56:35.719967 8684 catalog_manager.cc:4830] TS b3b45c9058d84b4e9ace42c9154ab0e0 (127.5.139.67:35389): tablet aa477b2bd34d4b74b245314148e4851d (table rebalance_test_table_2 [id=876faabf4df0421f8ee6dcd84e49c842]) successfully deleted
I20240501 19:56:35.801971 10232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:35.803575 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:35.804546 11712 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:35.805487 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:35.814364 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:35.823292 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:35.823298 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:35.826316 8949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:35.918000 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:35.920900 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:35.922820 10232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:35.925417 11712 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:35.927343 8949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:35.933653 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:35.945941 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:35.954155 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:36.042236 10495 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:36.043962 8819 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 19:56:36.047603 10232 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:36.049121 11712 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:36.058527 8949 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:36.061183 10363 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:36.063222 9080 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:36.069368 10101 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
2ee3823193724caab53a0a77e70e18df | 127.5.139.126:34427 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.139.84:42171 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
0f2b49c6d3dc49fabee40802e73ac982 | 127.5.139.65:38731 | HEALTHY | <none> | 3 | 0
14798909ff724444a8be03c45c41929a | 127.5.139.71:38393 | HEALTHY | <none> | 3 | 0
4663c01c9815427ab4bba05e1d1e44aa | 127.5.139.72:33401 | HEALTHY | <none> | 2 | 0
6977f9a52e9b4d06bcb12f9bee61553b | 127.5.139.69:42483 | HEALTHY | <none> | 0 | 0
6b4f42453a864314adc86aba2c3b3556 | 127.5.139.70:43441 | HEALTHY | <none> | 1 | 0
ac13bdb319c448dba072fb9e62d6893b | 127.5.139.66:43725 | HEALTHY | <none> | 2 | 0
b3b45c9058d84b4e9ace42c9154ab0e0 | 127.5.139.67:35389 | HEALTHY | <none> | 4 | 0
c9564f3b39b549c1bd995354fa135c5f | 127.5.139.68:46413 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.5.139.65 | experimental | 127.5.139.65:38731
local_ip_for_outbound_sockets | 127.5.139.66 | experimental | 127.5.139.66:43725
local_ip_for_outbound_sockets | 127.5.139.67 | experimental | 127.5.139.67:35389
local_ip_for_outbound_sockets | 127.5.139.68 | experimental | 127.5.139.68:46413
local_ip_for_outbound_sockets | 127.5.139.69 | experimental | 127.5.139.69:42483
local_ip_for_outbound_sockets | 127.5.139.70 | experimental | 127.5.139.70:43441
local_ip_for_outbound_sockets | 127.5.139.71 | experimental | 127.5.139.71:38393
local_ip_for_outbound_sockets | 127.5.139.72 | experimental | 127.5.139.72:33401
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.139.65:38731
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.139.66:43725
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.139.67:35389
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.5.139.68:46413
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.5.139.69:42483
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.5.139.70:43441
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.5.139.71:38393
server_dump_info_path | /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.5.139.72:33401
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+--------------------+-------------------------
builtin_ntp_servers | 127.5.139.84:42171 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 19:56:36.107089 5677 log_verifier.cc:126] Checking tablet 0c4e313b03cc44d0a2f7864f7e9d4f90
I20240501 19:56:36.156822 5677 log_verifier.cc:177] Verified matching terms for 277 ops in tablet 0c4e313b03cc44d0a2f7864f7e9d4f90
I20240501 19:56:36.156975 5677 log_verifier.cc:126] Checking tablet 0ca216ce0c6a420f8c90273a02ad8cf4
I20240501 19:56:36.204331 5677 log_verifier.cc:177] Verified matching terms for 257 ops in tablet 0ca216ce0c6a420f8c90273a02ad8cf4
I20240501 19:56:36.204471 5677 log_verifier.cc:126] Checking tablet 0f1ab4dc3bb841c2b352cee4ec7aa373
I20240501 19:56:36.249776 5677 log_verifier.cc:177] Verified matching terms for 258 ops in tablet 0f1ab4dc3bb841c2b352cee4ec7aa373
I20240501 19:56:36.249905 5677 log_verifier.cc:126] Checking tablet 3211e49e29174e0dbf667959a78c30e2
I20240501 19:56:36.291407 5677 log_verifier.cc:177] Verified matching terms for 221 ops in tablet 3211e49e29174e0dbf667959a78c30e2
I20240501 19:56:36.291554 5677 log_verifier.cc:126] Checking tablet 49f4147edb7349bcbf51994e59501d19
I20240501 19:56:36.348470 5677 log_verifier.cc:177] Verified matching terms for 322 ops in tablet 49f4147edb7349bcbf51994e59501d19
I20240501 19:56:36.348620 5677 log_verifier.cc:126] Checking tablet 58979b7e47734549bc48710ff1d2b1c1
I20240501 19:56:36.394665 5677 log_verifier.cc:177] Verified matching terms for 264 ops in tablet 58979b7e47734549bc48710ff1d2b1c1
I20240501 19:56:36.394799 5677 log_verifier.cc:126] Checking tablet 68c8311312bd4b50bdbedf414ada6bd2
I20240501 19:56:36.434780 5677 log_verifier.cc:177] Verified matching terms for 230 ops in tablet 68c8311312bd4b50bdbedf414ada6bd2
I20240501 19:56:36.434918 5677 log_verifier.cc:126] Checking tablet 6db903f127ce471d9cb87537a3cb7286
I20240501 19:56:36.480293 5677 log_verifier.cc:177] Verified matching terms for 250 ops in tablet 6db903f127ce471d9cb87537a3cb7286
I20240501 19:56:36.480540 5677 log_verifier.cc:126] Checking tablet 74706b357b2b43549757d24fa9a9d282
I20240501 19:56:36.526875 5677 log_verifier.cc:177] Verified matching terms for 270 ops in tablet 74706b357b2b43549757d24fa9a9d282
I20240501 19:56:36.527009 5677 log_verifier.cc:126] Checking tablet aa477b2bd34d4b74b245314148e4851d
I20240501 19:56:36.579180 5677 log_verifier.cc:177] Verified matching terms for 283 ops in tablet aa477b2bd34d4b74b245314148e4851d
I20240501 19:56:36.579324 5677 log_verifier.cc:126] Checking tablet ac9114fe81be4ba0bb4aadf28c61490d
I20240501 19:56:36.630854 5677 log_verifier.cc:177] Verified matching terms for 283 ops in tablet ac9114fe81be4ba0bb4aadf28c61490d
I20240501 19:56:36.631009 5677 log_verifier.cc:126] Checking tablet c63e353d2a544ce0af8551d8024743b7
I20240501 19:56:36.677639 5677 log_verifier.cc:177] Verified matching terms for 249 ops in tablet c63e353d2a544ce0af8551d8024743b7
I20240501 19:56:36.677775 5677 log_verifier.cc:126] Checking tablet c6e64c2bfdce4caf9c91618400c7f5d6
I20240501 19:56:36.691959 5677 log_verifier.cc:177] Verified matching terms for 5 ops in tablet c6e64c2bfdce4caf9c91618400c7f5d6
I20240501 19:56:36.692055 5677 log_verifier.cc:126] Checking tablet c869ad944b7a45d39ec171b0add850fb
I20240501 19:56:36.734180 5677 log_verifier.cc:177] Verified matching terms for 225 ops in tablet c869ad944b7a45d39ec171b0add850fb
I20240501 19:56:36.734292 5677 log_verifier.cc:126] Checking tablet d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:36.784898 5677 log_verifier.cc:177] Verified matching terms for 278 ops in tablet d827d0b0d5ba4969908768c96f92c1f3
I20240501 19:56:36.785054 5677 log_verifier.cc:126] Checking tablet df35daed2f72400086ccd0b8c47ecab1
I20240501 19:56:36.825459 5677 log_verifier.cc:177] Verified matching terms for 224 ops in tablet df35daed2f72400086ccd0b8c47ecab1
I20240501 19:56:36.826993 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 8757
I20240501 19:56:36.834859 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 8888
I20240501 19:56:36.842867 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 9019
I20240501 19:56:36.848469 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 10039
I20240501 19:56:36.853614 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 10170
I20240501 19:56:36.859544 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 10301
I20240501 19:56:36.865312 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 10432
I20240501 19:56:36.870755 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 11650
I20240501 19:56:36.876188 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 8666
2024-05-01T19:56:36Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (23013 ms)
[----------] 1 test from RebalanceParamTest (23013 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240501 19:56:36.931257 5677 test_util.cc:268] Using random seed: -497304909
I20240501 19:56:36.932013 5677 ts_itest-base.cc:112] Starting cluster with:
I20240501 19:56:36.932065 5677 ts_itest-base.cc:113] --------------
I20240501 19:56:36.932081 5677 ts_itest-base.cc:114] 8 tablet servers
I20240501 19:56:36.932093 5677 ts_itest-base.cc:115] 3 replicas per TS
I20240501 19:56:36.932109 5677 ts_itest-base.cc:116] --------------
2024-05-01T19:56:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T19:56:36Z Disabled control of system clock
I20240501 19:56:36.940657 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.139.126:35403
--webserver_interface=127.5.139.126
--webserver_port=0
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.139.126:35403
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 19:56:37.045702 12422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.045931 12422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:37.045971 12422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:37.048929 12422 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:37.048996 12422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 19:56:37.049021 12422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:37.049083 12422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 19:56:37.049114 12422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 19:56:37.052748 12422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.5.139.126:35403
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.139.126:35403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.139.126
--webserver_port=0
--never_fsync=true
--heap_profile_path=/tmp/kudu.12422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:37.053635 12422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:37.054742 12422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:37.063623 12428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.064294 12427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.064729 12430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.065451 12422 server_base.cc:1034] running on GCE node
I20240501 19:56:37.065979 12422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:37.066939 12422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:37.068096 12422 hybrid_clock.cc:648] HybridClock initialized: now 1714593397068083 us; error 31 us; skew 500 ppm
I20240501 19:56:37.070057 12422 webserver.cc:460] Webserver started at http://127.5.139.126:45489/ using document root <none> and password file <none>
I20240501 19:56:37.070600 12422 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:37.070658 12422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:37.070852 12422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:37.072453 12422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "be1305e487d64b748456a721443e3b3b"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.072940 12422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "be1305e487d64b748456a721443e3b3b"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.076220 12422 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:37.078363 12436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.079319 12422 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.079427 12422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
uuid: "be1305e487d64b748456a721443e3b3b"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.079516 12422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:37.130594 12422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:37.131244 12422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:37.131415 12422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:37.138242 12422 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.126:35403
I20240501 19:56:37.138252 12488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.126:35403 every 8 connection(s)
I20240501 19:56:37.139164 12422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 19:56:37.141577 12489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:37.146257 12489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b: Bootstrap starting.
I20240501 19:56:37.146387 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 12422
I20240501 19:56:37.146517 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 19:56:37.148586 12489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:37.149272 12489 log.cc:824] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:37.150866 12489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b: No bootstrap required, opened a new log
I20240501 19:56:37.153424 12489 raft_consensus.cc:348] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1305e487d64b748456a721443e3b3b" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35403 } }
I20240501 19:56:37.153637 12489 raft_consensus.cc:374] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:37.153697 12489 raft_consensus.cc:729] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be1305e487d64b748456a721443e3b3b, State: Initialized, Role: FOLLOWER
I20240501 19:56:37.154109 12489 consensus_queue.cc:260] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1305e487d64b748456a721443e3b3b" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35403 } }
I20240501 19:56:37.154255 12489 raft_consensus.cc:388] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 19:56:37.154305 12489 raft_consensus.cc:482] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 19:56:37.154381 12489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:37.154994 12489 raft_consensus.cc:504] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1305e487d64b748456a721443e3b3b" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35403 } }
I20240501 19:56:37.155525 12489 leader_election.cc:302] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be1305e487d64b748456a721443e3b3b; no voters:
I20240501 19:56:37.155889 12489 leader_election.cc:288] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 19:56:37.156544 12494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:37.157778 12494 raft_consensus.cc:686] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [term 1 LEADER]: Becoming Leader. State: Replica: be1305e487d64b748456a721443e3b3b, State: Running, Role: LEADER
I20240501 19:56:37.158077 12489 sys_catalog.cc:564] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [sys.catalog]: configured and running, proceeding with master startup.
I20240501 19:56:37.158532 12494 consensus_queue.cc:237] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1305e487d64b748456a721443e3b3b" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35403 } }
I20240501 19:56:37.160002 12495 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be1305e487d64b748456a721443e3b3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1305e487d64b748456a721443e3b3b" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35403 } } }
I20240501 19:56:37.159999 12496 sys_catalog.cc:455] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader be1305e487d64b748456a721443e3b3b. Latest consensus state: current_term: 1 leader_uuid: "be1305e487d64b748456a721443e3b3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be1305e487d64b748456a721443e3b3b" member_type: VOTER last_known_addr { host: "127.5.139.126" port: 35403 } } }
I20240501 19:56:37.160149 12496 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [sys.catalog]: This master's current role is: LEADER
I20240501 19:56:37.160149 12495 sys_catalog.cc:458] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b [sys.catalog]: This master's current role is: LEADER
W20240501 19:56:37.162498 12508 catalog_manager.cc:1558] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240501 19:56:37.162562 12508 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240501 19:56:37.162638 12509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 19:56:37.163264 12509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 19:56:37.167522 12509 catalog_manager.cc:1347] Generated new cluster ID: 16676184bece45f79372024b2abe1c6e
I20240501 19:56:37.167577 12509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 19:56:37.180837 12509 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 19:56:37.181943 12509 catalog_manager.cc:1504] Loading token signing keys...
I20240501 19:56:37.188030 12509 catalog_manager.cc:5857] T 00000000000000000000000000000000 P be1305e487d64b748456a721443e3b3b: Generated new TSK 0
I20240501 19:56:37.188741 12509 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 19:56:37.201167 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.65:0
--local_ip_for_outbound_sockets=127.5.139.65
--webserver_interface=127.5.139.65
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:37.305104 12513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.305311 12513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:37.305356 12513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:37.308395 12513 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:37.308465 12513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:37.308537 12513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.65
I20240501 19:56:37.312199 12513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.139.65
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.12513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.65
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:37.313171 12513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:37.314276 12513 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:37.322712 12519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.323323 12518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.323945 12521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.324604 12513 server_base.cc:1034] running on GCE node
I20240501 19:56:37.325013 12513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:37.325587 12513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:37.326753 12513 hybrid_clock.cc:648] HybridClock initialized: now 1714593397326739 us; error 34 us; skew 500 ppm
I20240501 19:56:37.328667 12513 webserver.cc:460] Webserver started at http://127.5.139.65:38255/ using document root <none> and password file <none>
I20240501 19:56:37.329308 12513 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:37.329373 12513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:37.329561 12513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:37.331247 12513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.331715 12513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.335122 12513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:37.337416 12527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.338485 12513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:37.338603 12513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.338708 12513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:37.370595 12513 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:37.371214 12513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:37.371384 12513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:37.371908 12513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:37.372771 12513 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:37.372818 12513 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.372864 12513 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:37.372895 12513 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.381392 12513 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.65:38745
I20240501 19:56:37.381438 12640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.65:38745 every 8 connection(s)
I20240501 19:56:37.382283 12513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 19:56:37.386981 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 12513
I20240501 19:56:37.387089 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 19:56:37.389371 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.66:0
--local_ip_for_outbound_sockets=127.5.139.66
--webserver_interface=127.5.139.66
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:37.391443 12641 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:37.391664 12641 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:37.392148 12641 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:37.393533 12453 ts_manager.cc:194] Registered new tserver with Master: ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:37.394658 12453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.65:39485
W20240501 19:56:37.492288 12644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.492508 12644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:37.492576 12644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:37.495548 12644 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:37.495612 12644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:37.495703 12644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.66
I20240501 19:56:37.499127 12644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.139.66
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.12644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.66
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:37.499909 12644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:37.500962 12644 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:37.509274 12652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.509841 12649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.510321 12650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.510969 12644 server_base.cc:1034] running on GCE node
I20240501 19:56:37.511322 12644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:37.511834 12644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:37.512995 12644 hybrid_clock.cc:648] HybridClock initialized: now 1714593397512968 us; error 42 us; skew 500 ppm
I20240501 19:56:37.514899 12644 webserver.cc:460] Webserver started at http://127.5.139.66:46447/ using document root <none> and password file <none>
I20240501 19:56:37.515432 12644 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:37.515492 12644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:37.515659 12644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:37.517287 12644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.517750 12644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.520844 12644 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 19:56:37.522939 12658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.523916 12644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.524011 12644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.524089 12644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:37.539068 12644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:37.539587 12644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:37.539736 12644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:37.540218 12644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:37.541087 12644 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:37.541134 12644 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.541188 12644 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:37.541215 12644 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.549274 12644 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.66:36895
I20240501 19:56:37.549299 12771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.66:36895 every 8 connection(s)
I20240501 19:56:37.550146 12644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 19:56:37.554888 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 12644
I20240501 19:56:37.555023 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 19:56:37.557161 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.67:0
--local_ip_for_outbound_sockets=127.5.139.67
--webserver_interface=127.5.139.67
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:37.558579 12772 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:37.558801 12772 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:37.559444 12772 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:37.560295 12453 ts_manager.cc:194] Registered new tserver with Master: 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:37.561097 12453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.66:36479
W20240501 19:56:37.659422 12775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.659622 12775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:37.659663 12775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:37.662696 12775 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:37.662757 12775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:37.662839 12775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.67
I20240501 19:56:37.666317 12775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.139.67
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.12775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.67
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:37.667140 12775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:37.668176 12775 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:37.675976 12783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.676683 12780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.677253 12781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.677987 12775 server_base.cc:1034] running on GCE node
I20240501 19:56:37.678388 12775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:37.678972 12775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:37.680132 12775 hybrid_clock.cc:648] HybridClock initialized: now 1714593397680119 us; error 24 us; skew 500 ppm
I20240501 19:56:37.682034 12775 webserver.cc:460] Webserver started at http://127.5.139.67:35967/ using document root <none> and password file <none>
I20240501 19:56:37.682572 12775 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:37.682668 12775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:37.682864 12775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:37.684527 12775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.684989 12775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.688431 12775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:37.690702 12789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.691658 12775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:37.691751 12775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.691823 12775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:37.705520 12775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:37.706050 12775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:37.706174 12775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:37.706615 12775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:37.707423 12775 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:37.707468 12775 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.707506 12775 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:37.707535 12775 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.715540 12775 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.67:46467
I20240501 19:56:37.715590 12902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.67:46467 every 8 connection(s)
I20240501 19:56:37.716411 12775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 19:56:37.723664 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 12775
I20240501 19:56:37.723798 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 19:56:37.724691 12903 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:37.724879 12903 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:37.725410 12903 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:37.725739 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.68:0
--local_ip_for_outbound_sockets=127.5.139.68
--webserver_interface=127.5.139.68
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:37.726341 12453 ts_manager.cc:194] Registered new tserver with Master: a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:37.727110 12453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.67:35569
W20240501 19:56:37.829154 12906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.829346 12906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:37.829387 12906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:37.832428 12906 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:37.832489 12906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:37.832551 12906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.68
I20240501 19:56:37.835981 12906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.139.68
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.12906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.68
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:37.836743 12906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:37.837843 12906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:37.845975 12912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.846493 12914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:37.846979 12911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:37.847681 12906 server_base.cc:1034] running on GCE node
I20240501 19:56:37.848037 12906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:37.848549 12906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:37.849700 12906 hybrid_clock.cc:648] HybridClock initialized: now 1714593397849686 us; error 24 us; skew 500 ppm
I20240501 19:56:37.851487 12906 webserver.cc:460] Webserver started at http://127.5.139.68:36717/ using document root <none> and password file <none>
I20240501 19:56:37.852010 12906 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:37.852069 12906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:37.852234 12906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:37.853901 12906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "2d52575d9e1b40dca0fc553d7295eb74"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.854355 12906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "2d52575d9e1b40dca0fc553d7295eb74"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.857481 12906 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:37.859527 12920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.860458 12906 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:37.860544 12906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "2d52575d9e1b40dca0fc553d7295eb74"
format_stamp: "Formatted at 2024-05-01 19:56:37 on dist-test-slave-twc7"
I20240501 19:56:37.860611 12906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:37.879093 12906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:37.879626 12906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:37.879755 12906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:37.880199 12906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:37.880995 12906 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:37.881058 12906 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.881114 12906 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:37.881139 12906 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:37.889289 12906 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.68:41997
I20240501 19:56:37.889346 13033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.68:41997 every 8 connection(s)
I20240501 19:56:37.890133 12906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 19:56:37.892341 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 12906
I20240501 19:56:37.892478 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 19:56:37.894757 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.69:0
--local_ip_for_outbound_sockets=127.5.139.69
--webserver_interface=127.5.139.69
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:37.899268 13034 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:37.899464 13034 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:37.899936 13034 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:37.900843 12453 ts_manager.cc:194] Registered new tserver with Master: 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:37.901391 12453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.68:50717
W20240501 19:56:37.997839 13037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:37.998059 13037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:37.998129 13037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:38.001109 13037 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:38.001173 13037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:38.001264 13037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.69
I20240501 19:56:38.004679 13037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.139.69
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.13037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.69
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:38.005537 13037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:38.006590 13037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:38.015575 13042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.016224 13045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.016757 13043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:38.017395 13037 server_base.cc:1034] running on GCE node
I20240501 19:56:38.017774 13037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:38.018321 13037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:38.019474 13037 hybrid_clock.cc:648] HybridClock initialized: now 1714593398019463 us; error 31 us; skew 500 ppm
I20240501 19:56:38.021449 13037 webserver.cc:460] Webserver started at http://127.5.139.69:40195/ using document root <none> and password file <none>
I20240501 19:56:38.021987 13037 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:38.022056 13037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:38.022231 13037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:38.023911 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9069ec6fcdb248479037f2d37e0e851d"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.024371 13037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9069ec6fcdb248479037f2d37e0e851d"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.027638 13037 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:38.029805 13051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.030890 13037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 19:56:38.030984 13037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9069ec6fcdb248479037f2d37e0e851d"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.031069 13037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:38.070344 13037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:38.070961 13037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:38.071097 13037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:38.071732 13037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:38.072587 13037 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:38.072635 13037 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.072676 13037 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:38.072700 13037 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.080993 13037 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.69:34719
I20240501 19:56:38.081058 13164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.69:34719 every 8 connection(s)
I20240501 19:56:38.081900 13037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:38.090056 13165 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:38.090243 13165 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:38.090703 13165 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:38.090875 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 13037
I20240501 19:56:38.090992 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 19:56:38.091666 12453 ts_manager.cc:194] Registered new tserver with Master: 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719)
I20240501 19:56:38.092368 12453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.69:60195
I20240501 19:56:38.093446 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.70:0
--local_ip_for_outbound_sockets=127.5.139.70
--webserver_interface=127.5.139.70
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:38.195957 13168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:38.196148 13168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:38.196216 13168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:38.198946 13168 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:38.199002 13168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:38.199115 13168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.70
I20240501 19:56:38.202417 13168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.139.70
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.13168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.70
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:38.203169 13168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:38.204174 13168 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:38.212047 13173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.212576 13174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.213191 13176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:38.213848 13168 server_base.cc:1034] running on GCE node
I20240501 19:56:38.214185 13168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:38.214650 13168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:38.215829 13168 hybrid_clock.cc:648] HybridClock initialized: now 1714593398215820 us; error 28 us; skew 500 ppm
I20240501 19:56:38.217612 13168 webserver.cc:460] Webserver started at http://127.5.139.70:39599/ using document root <none> and password file <none>
I20240501 19:56:38.218133 13168 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:38.218189 13168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:38.218361 13168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:38.219882 13168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "2ca06dad25a645bca71b09741798194a"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.220327 13168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "2ca06dad25a645bca71b09741798194a"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.223476 13168 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20240501 19:56:38.225572 13182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.226466 13168 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:38.226563 13168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2ca06dad25a645bca71b09741798194a"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.226637 13168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:38.240475 13168 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:38.240974 13168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:38.241133 13168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:38.241601 13168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:38.242411 13168 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:38.242466 13168 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.242517 13168 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:38.242538 13168 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.250335 13168 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.70:33933
I20240501 19:56:38.250380 13295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.70:33933 every 8 connection(s)
I20240501 19:56:38.251184 13168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:38.258762 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 13168
I20240501 19:56:38.258934 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 19:56:38.259644 13296 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:38.259863 13296 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:38.260411 13296 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:38.261063 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.71:0
--local_ip_for_outbound_sockets=127.5.139.71
--webserver_interface=127.5.139.71
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:38.262436 12453 ts_manager.cc:194] Registered new tserver with Master: 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933)
I20240501 19:56:38.262966 12453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.70:53827
W20240501 19:56:38.364614 13299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:38.364853 13299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:38.365074 13299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:38.368022 13299 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:38.368081 13299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:38.368142 13299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.71
I20240501 19:56:38.371538 13299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.139.71
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.13299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.71
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:38.372269 13299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:38.373281 13299 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:38.381161 13307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.381711 13304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.382200 13305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:38.382834 13299 server_base.cc:1034] running on GCE node
I20240501 19:56:38.383178 13299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:38.383762 13299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:38.384954 13299 hybrid_clock.cc:648] HybridClock initialized: now 1714593398384934 us; error 37 us; skew 500 ppm
I20240501 19:56:38.386744 13299 webserver.cc:460] Webserver started at http://127.5.139.71:38429/ using document root <none> and password file <none>
I20240501 19:56:38.387257 13299 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:38.387313 13299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:38.387475 13299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:38.388994 13299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "1d49e80be3d04fb99b711c62d61bac09"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.389525 13299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "1d49e80be3d04fb99b711c62d61bac09"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.392988 13299 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 19:56:38.395188 13313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.396175 13299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:38.396288 13299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1d49e80be3d04fb99b711c62d61bac09"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.396389 13299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:38.396658 12641 heartbeater.cc:503] Master 127.5.139.126:35403 was elected leader, sending a full tablet report...
I20240501 19:56:38.412691 13299 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:38.413280 13299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:38.413431 13299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:38.413911 13299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:38.414737 13299 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:38.414785 13299 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.414839 13299 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:38.414870 13299 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.422895 13299 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.71:46083
I20240501 19:56:38.422966 13426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.71:46083 every 8 connection(s)
I20240501 19:56:38.423802 13299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:38.426789 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 13299
I20240501 19:56:38.426934 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 19:56:38.429217 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.72:0
--local_ip_for_outbound_sockets=127.5.139.72
--webserver_interface=127.5.139.72
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:38.433431 13427 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:38.433637 13427 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:38.434141 13427 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:38.435045 12453 ts_manager.cc:194] Registered new tserver with Master: 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 19:56:38.435554 12453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.71:33867
W20240501 19:56:38.534788 13430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:38.534998 13430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:38.535065 13430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:38.538094 13430 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:38.538161 13430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:38.538249 13430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.72
I20240501 19:56:38.541734 13430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.139.72
--webserver_port=0
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.13430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.72
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:38.542573 13430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:38.543663 13430 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:38.551815 13435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.552371 13436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:38.552917 13438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:38.553567 13430 server_base.cc:1034] running on GCE node
I20240501 19:56:38.553947 13430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:38.554526 13430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:38.555709 13430 hybrid_clock.cc:648] HybridClock initialized: now 1714593398555670 us; error 39 us; skew 500 ppm
I20240501 19:56:38.557549 13430 webserver.cc:460] Webserver started at http://127.5.139.72:38791/ using document root <none> and password file <none>
I20240501 19:56:38.558048 13430 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:38.558109 13430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:38.558313 13430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 19:56:38.559931 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "c6d2c7096e404ea9a6f73b123d74c622"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.560396 13430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "c6d2c7096e404ea9a6f73b123d74c622"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.562885 12772 heartbeater.cc:503] Master 127.5.139.126:35403 was elected leader, sending a full tablet report...
I20240501 19:56:38.563587 13430 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20240501 19:56:38.565914 13444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.566839 13430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:38.566929 13430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c6d2c7096e404ea9a6f73b123d74c622"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:38.567018 13430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:38.590116 13430 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:38.590662 13430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:38.590813 13430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:38.591295 13430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:38.592111 13430 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:38.592159 13430 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.592200 13430 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:38.592233 13430 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:38.600561 13430 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.72:37453
I20240501 19:56:38.600630 13557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.72:37453 every 8 connection(s)
I20240501 19:56:38.601469 13430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:38.605619 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 13430
I20240501 19:56:38.605792 5677 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 19:56:38.609820 13558 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:38.610003 13558 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:38.610488 13558 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:38.611385 12453 ts_manager.cc:194] Registered new tserver with Master: c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
I20240501 19:56:38.611881 12453 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.72:51569
I20240501 19:56:38.619714 5677 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:56:38.628669 5677 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 19:56:38.628738 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 13037
I20240501 19:56:38.634236 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 13168
I20240501 19:56:38.639326 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 13299
I20240501 19:56:38.644716 5677 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskuLQw07/build/debug/bin/kudu with pid 13430
I20240501 19:56:38.729126 12903 heartbeater.cc:503] Master 127.5.139.126:35403 was elected leader, sending a full tablet report...
I20240501 19:56:38.903136 13034 heartbeater.cc:503] Master 127.5.139.126:35403 was elected leader, sending a full tablet report...
I20240501 19:56:42.401345 12453 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:42.420320 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 04a70a2ee2144f2287795ffe723d2716 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.421030 12837 tablet_service.cc:1467] Processing CreateTablet for tablet db240ec5f8b2444fb6114ad814f3cdb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.421481 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a70a2ee2144f2287795ffe723d2716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.421335 12573 tablet_service.cc:1467] Processing CreateTablet for tablet 04a70a2ee2144f2287795ffe723d2716 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.421914 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db240ec5f8b2444fb6114ad814f3cdb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.422070 12573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a70a2ee2144f2287795ffe723d2716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.421335 12574 tablet_service.cc:1467] Processing CreateTablet for tablet db240ec5f8b2444fb6114ad814f3cdb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.420320 12968 tablet_service.cc:1467] Processing CreateTablet for tablet 12a50be843fb49ee8a66c9a17d271d61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.422441 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a50be843fb49ee8a66c9a17d271d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.423082 12574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db240ec5f8b2444fb6114ad814f3cdb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.425289 12575 tablet_service.cc:1467] Processing CreateTablet for tablet 12a50be843fb49ee8a66c9a17d271d61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.425508 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a50be843fb49ee8a66c9a17d271d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.426672 12836 tablet_service.cc:1467] Processing CreateTablet for tablet 04a70a2ee2144f2287795ffe723d2716 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.426944 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a70a2ee2144f2287795ffe723d2716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.428022 13579 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:42.428668 13578 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:42.430104 13579 tablet_bootstrap.cc:654] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.430760 13578 tablet_bootstrap.cc:654] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.430941 13579 log.cc:824] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:42.431509 13578 log.cc:824] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:42.433221 13578 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:42.433468 13579 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:42.433501 13578 ts_tablet_manager.cc:1367] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:42.433624 13579 ts_tablet_manager.cc:1367] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 19:56:42.433647 13581 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:42.434863 12705 tablet_service.cc:1467] Processing CreateTablet for tablet db240ec5f8b2444fb6114ad814f3cdb7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.434863 12706 tablet_service.cc:1467] Processing CreateTablet for tablet 12a50be843fb49ee8a66c9a17d271d61 (DEFAULT_TABLE table=rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.435761 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db240ec5f8b2444fb6114ad814f3cdb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.436048 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a50be843fb49ee8a66c9a17d271d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.437543 13578 raft_consensus.cc:348] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:42.437850 13578 raft_consensus.cc:374] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.437951 13578 raft_consensus.cc:729] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.437888 13579 raft_consensus.cc:348] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.438076 13579 raft_consensus.cc:374] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.438129 13579 raft_consensus.cc:729] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.438555 13579 consensus_queue.cc:260] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.438555 13578 consensus_queue.cc:260] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:42.439373 13578 ts_tablet_manager.cc:1398] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:56:42.439373 13579 ts_tablet_manager.cc:1398] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20240501 19:56:42.439692 13579 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:42.440548 13579 tablet_bootstrap.cc:654] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.441597 13578 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:42.441632 13579 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:42.441705 13579 ts_tablet_manager.cc:1367] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.442227 13579 raft_consensus.cc:348] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:42.442384 13579 raft_consensus.cc:374] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.442425 13579 raft_consensus.cc:729] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.442540 13579 consensus_queue.cc:260] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:42.442865 13578 tablet_bootstrap.cc:654] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.443109 13579 ts_tablet_manager.cc:1398] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 19:56:42.444063 13578 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:42.444192 13578 ts_tablet_manager.cc:1367] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:42.444620 13578 raft_consensus.cc:348] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.444715 13578 raft_consensus.cc:374] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.444769 13578 raft_consensus.cc:729] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.444881 13578 consensus_queue.cc:260] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.445173 13578 ts_tablet_manager.cc:1398] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.445305 13581 tablet_bootstrap.cc:654] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.445606 13585 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:42.446056 13581 log.cc:824] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:42.447386 13585 tablet_bootstrap.cc:654] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.447824 13581 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:42.447995 13581 ts_tablet_manager.cc:1367] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240501 19:56:42.448057 13585 log.cc:824] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:42.449618 13585 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:42.449776 13585 ts_tablet_manager.cc:1367] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:42.451269 13581 raft_consensus.cc:348] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:42.451517 13581 raft_consensus.cc:374] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.451575 13581 raft_consensus.cc:729] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.452014 13581 consensus_queue.cc:260] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:42.452817 13581 ts_tablet_manager.cc:1398] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240501 19:56:42.453119 13581 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:42.454142 13581 tablet_bootstrap.cc:654] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.454089 13585 raft_consensus.cc:348] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.455307 13585 raft_consensus.cc:374] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.455370 13585 raft_consensus.cc:729] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.455499 13581 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:42.455667 13581 ts_tablet_manager.cc:1367] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:42.455781 13585 consensus_queue.cc:260] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.456204 13581 raft_consensus.cc:348] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.456313 13581 raft_consensus.cc:374] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.456346 13581 raft_consensus.cc:729] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.456451 13581 consensus_queue.cc:260] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.456605 13585 ts_tablet_manager.cc:1398] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20240501 19:56:42.456703 13581 ts_tablet_manager.cc:1398] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:42.456979 13581 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:42.457836 13585 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:42.457908 13581 tablet_bootstrap.cc:654] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.458724 13585 tablet_bootstrap.cc:654] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.459143 13581 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:42.459219 13581 ts_tablet_manager.cc:1367] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.459672 13581 raft_consensus.cc:348] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.459815 13585 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:42.459769 13581 raft_consensus.cc:374] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.459858 13581 raft_consensus.cc:729] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.459892 13585 ts_tablet_manager.cc:1367] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:42.460001 13581 consensus_queue.cc:260] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.460347 13581 ts_tablet_manager.cc:1398] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.460389 13585 raft_consensus.cc:348] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.460459 13585 raft_consensus.cc:374] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.460489 13585 raft_consensus.cc:729] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.460597 13585 consensus_queue.cc:260] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.460829 13585 ts_tablet_manager.cc:1398] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20240501 19:56:42.470696 12904 tablet.cc:2369] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:42.554558 12773 tablet.cc:2369] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:42.607836 13583 raft_consensus.cc:482] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.608042 13583 raft_consensus.cc:504] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:42.609198 13583 leader_election.cc:288] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:42.612911 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a70a2ee2144f2287795ffe723d2716" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:42.612911 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a70a2ee2144f2287795ffe723d2716" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:42.613229 12595 raft_consensus.cc:2454] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:42.613229 12988 raft_consensus.cc:2454] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:42.613714 12792 leader_election.cc:302] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:42.613952 13583 raft_consensus.cc:2781] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.614028 13583 raft_consensus.cc:482] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.614079 13583 raft_consensus.cc:3037] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.614948 13583 raft_consensus.cc:504] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:42.615264 13583 leader_election.cc:288] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Requested vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:42.615571 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a70a2ee2144f2287795ffe723d2716" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:42.615583 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04a70a2ee2144f2287795ffe723d2716" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:42.615675 12988 raft_consensus.cc:3037] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.615689 12595 raft_consensus.cc:3037] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.616499 12988 raft_consensus.cc:2454] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:42.616789 12793 leader_election.cc:302] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:42.616958 12595 raft_consensus.cc:2454] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:42.616992 13583 raft_consensus.cc:2781] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.617131 13583 raft_consensus.cc:686] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Becoming Leader. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Running, Role: LEADER
I20240501 19:56:42.617465 13583 consensus_queue.cc:237] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:42.620393 12452 catalog_manager.cc:5484] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: term changed from 0 to 1, leader changed from <none> to a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67). New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:42.637005 12642 tablet.cc:2369] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 19:56:42.644057 13035 tablet.cc:2369] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 19:56:42.670815 13591 raft_consensus.cc:482] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.670929 13591 raft_consensus.cc:504] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.671872 13591 leader_election.cc:288] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:42.676565 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db240ec5f8b2444fb6114ad814f3cdb7" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:42.676582 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db240ec5f8b2444fb6114ad814f3cdb7" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:42.676738 12857 raft_consensus.cc:2454] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:42.676836 12726 raft_consensus.cc:2454] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:42.677138 12528 leader_election.cc:302] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:42.677356 13591 raft_consensus.cc:2781] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.677414 13591 raft_consensus.cc:482] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.677457 13591 raft_consensus.cc:3037] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.678282 13591 raft_consensus.cc:504] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.678601 13591 leader_election.cc:288] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:42.678951 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db240ec5f8b2444fb6114ad814f3cdb7" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:42.679006 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db240ec5f8b2444fb6114ad814f3cdb7" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:42.679081 12857 raft_consensus.cc:3037] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.679116 12726 raft_consensus.cc:3037] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.679841 12857 raft_consensus.cc:2454] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:42.679996 12726 raft_consensus.cc:2454] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:42.680132 12528 leader_election.cc:302] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:42.680310 13591 raft_consensus.cc:2781] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.680501 13591 raft_consensus.cc:686] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Becoming Leader. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:42.680810 13591 consensus_queue.cc:237] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.681475 13593 raft_consensus.cc:482] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:42.681557 13593 raft_consensus.cc:504] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.682428 13593 leader_election.cc:288] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:42.683890 12452 catalog_manager.cc:5484] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:42.686111 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a50be843fb49ee8a66c9a17d271d61" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:42.686234 12988 raft_consensus.cc:2454] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:42.686857 12662 leader_election.cc:302] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:42.687105 13593 raft_consensus.cc:2781] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:42.687155 13593 raft_consensus.cc:482] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:42.687192 13593 raft_consensus.cc:3037] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.687891 13593 raft_consensus.cc:504] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.688179 13593 leader_election.cc:288] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:42.688582 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a50be843fb49ee8a66c9a17d271d61" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:42.688704 12988 raft_consensus.cc:3037] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.689644 12988 raft_consensus.cc:2454] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:42.689579 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a50be843fb49ee8a66c9a17d271d61" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:42.689692 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12a50be843fb49ee8a66c9a17d271d61" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:42.689774 12595 raft_consensus.cc:2454] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:42.690042 12662 leader_election.cc:302] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:42.690203 13593 raft_consensus.cc:2781] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:42.690389 13593 raft_consensus.cc:686] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:42.690640 13593 consensus_queue.cc:237] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.693378 12452 catalog_manager.cc:5484] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:42.808310 13607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:42.808542 13607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:42.811432 13607 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 19:56:42.892544 12594 raft_consensus.cc:3037] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:42.893700 12594 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.894671 12988 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.895431 13593 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.895542 12726 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.895653 13605 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.896126 13591 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.897435 12857 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.901937 13601 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.902849 13641 mvcc.cc:205] Tried to move back new op lower bound from 7022974578254475264 to 7022974577383403520. Current Snapshot: MvccSnapshot[applied={T|T < 7022974578254475264}]
I20240501 19:56:42.903911 13639 mvcc.cc:205] Tried to move back new op lower bound from 7022974578254475264 to 7022974577383403520. Current Snapshot: MvccSnapshot[applied={T|T < 7022974578254475264}]
I20240501 19:56:42.909962 12987 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.910382 13583 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.911509 12595 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:42.912317 13583 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:42.914458 13646 mvcc.cc:205] Tried to move back new op lower bound from 7022974578254475264 to 7022974577383403520. Current Snapshot: MvccSnapshot[applied={T|T < 7022974578254475264}]
I20240501 19:56:42.924350 13643 mvcc.cc:205] Tried to move back new op lower bound from 7022974578240995328 to 7022974577422905344. Current Snapshot: MvccSnapshot[applied={T|T < 7022974578240995328}]
INSERT report
rows total: 2000
time total: 113.404 ms
time per row: 0.0567019 ms
I20240501 19:56:42.965163 12452 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:42.970098 12836 tablet_service.cc:1467] Processing CreateTablet for tablet 3e84a3865ce64eaebad4df46ec01842d (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.970127 12968 tablet_service.cc:1467] Processing CreateTablet for tablet 3e84a3865ce64eaebad4df46ec01842d (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.970399 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e84a3865ce64eaebad4df46ec01842d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.970396 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e84a3865ce64eaebad4df46ec01842d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.970396 12967 tablet_service.cc:1467] Processing CreateTablet for tablet e4b15761177f4cffb7aa5e4d16e42ac7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.972229 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b15761177f4cffb7aa5e4d16e42ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.972259 12705 tablet_service.cc:1467] Processing CreateTablet for tablet 3e84a3865ce64eaebad4df46ec01842d (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.972533 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e84a3865ce64eaebad4df46ec01842d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.972069 13669 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:42.974316 13671 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:42.974684 12573 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4d79f5e11b4923acb2ce543884b54a (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.974802 13670 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:42.974960 12573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4d79f5e11b4923acb2ce543884b54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.974819 12705 tablet_service.cc:1467] Processing CreateTablet for tablet e4b15761177f4cffb7aa5e4d16e42ac7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.975235 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b15761177f4cffb7aa5e4d16e42ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.975651 13670 tablet_bootstrap.cc:654] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.975805 13671 tablet_bootstrap.cc:654] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.974884 12706 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4d79f5e11b4923acb2ce543884b54a (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.977207 13671 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:42.977298 13671 ts_tablet_manager.cc:1367] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:42.977399 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4d79f5e11b4923acb2ce543884b54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.977586 12836 tablet_service.cc:1467] Processing CreateTablet for tablet e4b15761177f4cffb7aa5e4d16e42ac7 (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.977607 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 7e4d79f5e11b4923acb2ce543884b54a (DEFAULT_TABLE table=rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:42.977808 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b15761177f4cffb7aa5e4d16e42ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.977752 13671 raft_consensus.cc:348] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.977850 13671 raft_consensus.cc:374] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.977881 13671 raft_consensus.cc:729] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.977900 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4d79f5e11b4923acb2ce543884b54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:42.977994 13671 consensus_queue.cc:260] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.978343 13671 ts_tablet_manager.cc:1398] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.978497 13671 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:42.979313 13671 tablet_bootstrap.cc:654] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.979948 13674 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:42.980289 13669 tablet_bootstrap.cc:654] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.980492 13670 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:42.980571 13670 ts_tablet_manager.cc:1367] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:42.980842 13674 tablet_bootstrap.cc:654] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.981015 13670 raft_consensus.cc:348] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.981187 13670 raft_consensus.cc:374] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.981225 13670 raft_consensus.cc:729] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.981357 13670 consensus_queue.cc:260] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.981653 13670 ts_tablet_manager.cc:1398] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.981667 13669 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:42.981743 13669 ts_tablet_manager.cc:1367] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20240501 19:56:42.981818 13670 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:42.982028 13674 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:42.982115 13674 ts_tablet_manager.cc:1367] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:42.982247 13669 raft_consensus.cc:348] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.982347 13669 raft_consensus.cc:374] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.982383 13669 raft_consensus.cc:729] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.982539 13669 consensus_queue.cc:260] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:42.982607 13670 tablet_bootstrap.cc:654] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.982592 13674 raft_consensus.cc:348] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.982693 13674 raft_consensus.cc:374] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.982733 13674 raft_consensus.cc:729] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.982896 13669 ts_tablet_manager.cc:1398] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.982864 13674 consensus_queue.cc:260] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.983060 13669 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:42.983263 13674 ts_tablet_manager.cc:1398] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.983845 13669 tablet_bootstrap.cc:654] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.985337 13669 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:42.985414 13669 ts_tablet_manager.cc:1367] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.985824 13669 raft_consensus.cc:348] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:42.985934 13669 raft_consensus.cc:374] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.985947 13670 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:42.985968 13669 raft_consensus.cc:729] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.986002 13670 ts_tablet_manager.cc:1367] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:42.986078 13669 consensus_queue.cc:260] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:42.986347 13669 ts_tablet_manager.cc:1398] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:42.986400 13670 raft_consensus.cc:348] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:42.986511 13670 raft_consensus.cc:374] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.986555 13670 raft_consensus.cc:729] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.986685 13670 consensus_queue.cc:260] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:42.986950 13670 ts_tablet_manager.cc:1398] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.987092 13670 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:42.987731 13670 tablet_bootstrap.cc:654] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.988740 13670 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:42.988826 13670 ts_tablet_manager.cc:1367] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:42.989282 13671 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:42.989476 13671 ts_tablet_manager.cc:1367] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240501 19:56:42.989852 13670 raft_consensus.cc:348] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.989884 13671 raft_consensus.cc:348] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:42.989956 13670 raft_consensus.cc:374] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.989965 13671 raft_consensus.cc:374] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.990000 13670 raft_consensus.cc:729] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.990000 13671 raft_consensus.cc:729] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.990114 13670 consensus_queue.cc:260] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.990114 13671 consensus_queue.cc:260] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:42.990403 13670 ts_tablet_manager.cc:1398] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:42.990634 13671 ts_tablet_manager.cc:1398] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:42.990801 13671 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:42.991616 13671 tablet_bootstrap.cc:654] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:42.992998 13671 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:42.993109 13671 ts_tablet_manager.cc:1367] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:42.993429 13671 raft_consensus.cc:348] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.993506 13671 raft_consensus.cc:374] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:42.993543 13671 raft_consensus.cc:729] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:42.993661 13671 consensus_queue.cc:260] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:42.993885 13671 ts_tablet_manager.cc:1398] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:43.037591 13605 raft_consensus.cc:482] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.037720 13605 raft_consensus.cc:504] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:43.038230 13605 leader_election.cc:288] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:43.038296 13593 raft_consensus.cc:482] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.038378 13593 raft_consensus.cc:504] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.038709 13593 leader_election.cc:288] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:43.038726 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:43.038918 12987 raft_consensus.cc:2454] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:43.039197 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:43.039291 12662 leader_election.cc:302] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:43.039383 12595 raft_consensus.cc:2454] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:43.039456 13593 raft_consensus.cc:2781] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.039505 13593 raft_consensus.cc:482] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.039552 13593 raft_consensus.cc:3037] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.039711 12661 leader_election.cc:302] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:43.039889 13605 raft_consensus.cc:2781] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.039937 13605 raft_consensus.cc:482] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.039971 13605 raft_consensus.cc:3037] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.040611 13593 raft_consensus.cc:504] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:43.040869 13605 raft_consensus.cc:504] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.040982 13593 leader_election.cc:288] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:43.041224 13605 leader_election.cc:288] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:43.041450 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:43.041536 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:43.041582 12987 raft_consensus.cc:3037] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.041631 12595 raft_consensus.cc:3037] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.042516 12987 raft_consensus.cc:2454] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:43.042584 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:43.042682 12595 raft_consensus.cc:2454] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:43.042724 12857 raft_consensus.cc:2454] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:43.042809 12854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:43.042932 12854 raft_consensus.cc:3037] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.043008 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:43.043125 12855 raft_consensus.cc:3037] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.043385 12661 leader_election.cc:302] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:43.043555 13605 raft_consensus.cc:2781] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.043646 13605 raft_consensus.cc:686] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:43.043814 13605 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.043918 12854 raft_consensus.cc:2454] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:43.044134 12662 leader_election.cc:302] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:43.044526 13605 raft_consensus.cc:2781] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.044606 12855 raft_consensus.cc:2454] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:43.044629 13605 raft_consensus.cc:686] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:43.042914 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:43.044744 13605 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:43.044857 12856 raft_consensus.cc:2362] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1. Re-sending same reply.
I20240501 19:56:43.045651 12452 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.048691 12452 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.065282 13605 raft_consensus.cc:482] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.065390 13605 raft_consensus.cc:504] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.065764 13605 leader_election.cc:288] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:43.066161 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:43.066285 12856 raft_consensus.cc:2454] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:43.066319 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:43.066478 12987 raft_consensus.cc:2454] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:43.066603 12659 leader_election.cc:302] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:43.066780 13605 raft_consensus.cc:2781] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.066854 13605 raft_consensus.cc:482] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.066901 13605 raft_consensus.cc:3037] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.067656 13605 raft_consensus.cc:504] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.067963 13605 leader_election.cc:288] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:43.068329 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:43.068347 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:43.068459 12987 raft_consensus.cc:3037] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.068459 12856 raft_consensus.cc:3037] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.069432 12987 raft_consensus.cc:2454] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:43.069432 12856 raft_consensus.cc:2454] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:43.069736 12662 leader_election.cc:302] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:43.069900 13605 raft_consensus.cc:2781] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.069983 13605 raft_consensus.cc:686] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:43.070223 13605 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.071854 12452 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:43.180434 13686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:43.180644 13686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:43.183566 13686 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 19:56:43.265583 12856 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.266044 12987 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.266891 13593 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.267082 13605 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.267576 12856 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.267608 12595 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.267596 12855 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.268260 12987 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.268321 13682 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.268620 13682 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.268884 13682 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.274363 13605 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 111.773 ms
time per row: 0.0558863 ms
I20240501 19:56:43.335080 12447 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:43.340075 12705 tablet_service.cc:1467] Processing CreateTablet for tablet 8b61e94fc7e541c8aa84957560508a31 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.340217 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 8b61e94fc7e541c8aa84957560508a31 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.340409 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b61e94fc7e541c8aa84957560508a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.340497 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b61e94fc7e541c8aa84957560508a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.341384 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb481d394034a8f85af613beb5c9ec8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.341660 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb481d394034a8f85af613beb5c9ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.342384 13671 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:43.342592 13669 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:43.342602 12573 tablet_service.cc:1467] Processing CreateTablet for tablet 8b61e94fc7e541c8aa84957560508a31 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.342878 12573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b61e94fc7e541c8aa84957560508a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.343252 13671 tablet_bootstrap.cc:654] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.343425 13669 tablet_bootstrap.cc:654] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.343991 13670 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:43.344229 12968 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7ea564ea504163a1e17561be9a5130 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.344466 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7ea564ea504163a1e17561be9a5130. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.344832 13670 tablet_bootstrap.cc:654] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.344833 13674 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:43.344990 12574 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb481d394034a8f85af613beb5c9ec8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.345207 12574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb481d394034a8f85af613beb5c9ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.345670 13674 tablet_bootstrap.cc:654] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.345938 13669 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:43.346012 13669 ts_tablet_manager.cc:1367] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20240501 19:56:43.346426 13669 raft_consensus.cc:348] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.346552 13669 raft_consensus.cc:374] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.346591 13669 raft_consensus.cc:729] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.346709 13669 consensus_queue.cc:260] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.346995 13669 ts_tablet_manager.cc:1398] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:43.347107 12575 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7ea564ea504163a1e17561be9a5130 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.347180 13669 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:43.347302 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7ea564ea504163a1e17561be9a5130. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.347424 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 0fb481d394034a8f85af613beb5c9ec8 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.347617 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb481d394034a8f85af613beb5c9ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.347965 13669 tablet_bootstrap.cc:654] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.348991 13669 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:43.349146 13669 ts_tablet_manager.cc:1367] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:43.349732 12705 tablet_service.cc:1467] Processing CreateTablet for tablet 7d7ea564ea504163a1e17561be9a5130 (DEFAULT_TABLE table=rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.349980 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7ea564ea504163a1e17561be9a5130. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.350162 13674 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:43.350234 13670 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:43.350245 13674 ts_tablet_manager.cc:1367] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:43.350317 13670 ts_tablet_manager.cc:1367] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20240501 19:56:43.350436 13671 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:43.350503 13671 ts_tablet_manager.cc:1367] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:43.350648 13674 raft_consensus.cc:348] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.350750 13674 raft_consensus.cc:374] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.350780 13674 raft_consensus.cc:729] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.350773 13670 raft_consensus.cc:348] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.350872 13670 raft_consensus.cc:374] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.350909 13670 raft_consensus.cc:729] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.350895 13671 raft_consensus.cc:348] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.350904 13674 consensus_queue.cc:260] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.350986 13671 raft_consensus.cc:374] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.351015 13671 raft_consensus.cc:729] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.351033 13670 consensus_queue.cc:260] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.351135 13671 consensus_queue.cc:260] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.351200 13674 ts_tablet_manager.cc:1398] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.351303 13670 ts_tablet_manager.cc:1398] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.351372 13674 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:43.351405 13671 ts_tablet_manager.cc:1398] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.352221 13674 tablet_bootstrap.cc:654] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.352254 13669 raft_consensus.cc:348] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.352421 13669 raft_consensus.cc:374] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.352486 13669 raft_consensus.cc:729] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.352605 13671 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:43.352636 13669 consensus_queue.cc:260] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.352967 13669 ts_tablet_manager.cc:1398] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:43.353194 13669 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:43.353513 13671 tablet_bootstrap.cc:654] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.354126 13669 tablet_bootstrap.cc:654] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.355522 13671 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:43.355583 13669 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:43.355605 13671 ts_tablet_manager.cc:1367] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:43.355654 13669 ts_tablet_manager.cc:1367] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:43.356036 13671 raft_consensus.cc:348] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.356066 13669 raft_consensus.cc:348] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.356134 13671 raft_consensus.cc:374] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.356158 13669 raft_consensus.cc:374] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.356171 13671 raft_consensus.cc:729] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.356189 13669 raft_consensus.cc:729] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.356297 13669 consensus_queue.cc:260] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.356290 13671 consensus_queue.cc:260] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.356604 13671 ts_tablet_manager.cc:1398] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.356608 13669 ts_tablet_manager.cc:1398] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:43.356882 13674 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:43.356966 13674 ts_tablet_manager.cc:1367] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 19:56:43.357412 13674 raft_consensus.cc:348] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.357507 13674 raft_consensus.cc:374] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.357540 13674 raft_consensus.cc:729] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.357662 13674 consensus_queue.cc:260] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.357935 13674 ts_tablet_manager.cc:1398] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.358114 13674 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:43.359027 13674 tablet_bootstrap.cc:654] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.360211 13674 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:43.360296 13674 ts_tablet_manager.cc:1367] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:43.360597 13674 raft_consensus.cc:348] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.360671 13674 raft_consensus.cc:374] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.360704 13674 raft_consensus.cc:729] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.360787 13674 consensus_queue.cc:260] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.361003 13674 ts_tablet_manager.cc:1398] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:43.519627 13583 raft_consensus.cc:482] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.519768 13583 raft_consensus.cc:504] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.520203 13583 leader_election.cc:288] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:43.520663 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb481d394034a8f85af613beb5c9ec8" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:43.520668 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb481d394034a8f85af613beb5c9ec8" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:43.520795 12595 raft_consensus.cc:2454] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:43.520825 12988 raft_consensus.cc:2454] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:43.521108 12792 leader_election.cc:302] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:43.521299 13583 raft_consensus.cc:2781] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.521363 13583 raft_consensus.cc:482] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.521392 13583 raft_consensus.cc:3037] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.522234 13583 raft_consensus.cc:504] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.522543 13583 leader_election.cc:288] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Requested vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:43.522976 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb481d394034a8f85af613beb5c9ec8" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:43.522998 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0fb481d394034a8f85af613beb5c9ec8" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:43.523110 12595 raft_consensus.cc:3037] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.523124 12988 raft_consensus.cc:3037] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.524017 12988 raft_consensus.cc:2454] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:43.524020 12595 raft_consensus.cc:2454] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:43.524304 12793 leader_election.cc:302] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:43.524487 13583 raft_consensus.cc:2781] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.524569 13583 raft_consensus.cc:686] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Becoming Leader. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Running, Role: LEADER
I20240501 19:56:43.524708 13583 consensus_queue.cc:237] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.526394 12451 catalog_manager.cc:5484] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: term changed from 0 to 1, leader changed from <none> to a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67). New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.530215 13712 raft_consensus.cc:482] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.530305 13712 raft_consensus.cc:504] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.530645 13712 leader_election.cc:288] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:43.530915 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b61e94fc7e541c8aa84957560508a31" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:43.530961 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b61e94fc7e541c8aa84957560508a31" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:43.531021 12988 raft_consensus.cc:2454] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:43.531097 12595 raft_consensus.cc:2454] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:43.531270 12662 leader_election.cc:302] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:43.531458 13712 raft_consensus.cc:2781] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.531514 13712 raft_consensus.cc:482] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.531551 13712 raft_consensus.cc:3037] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.532284 13712 raft_consensus.cc:504] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.532560 13712 leader_election.cc:288] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:43.532861 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b61e94fc7e541c8aa84957560508a31" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:43.532871 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b61e94fc7e541c8aa84957560508a31" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:43.532945 12595 raft_consensus.cc:3037] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.532996 12988 raft_consensus.cc:3037] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.533742 12595 raft_consensus.cc:2454] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:43.533989 12988 raft_consensus.cc:2454] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:43.534078 12661 leader_election.cc:302] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:43.534267 13712 raft_consensus.cc:2781] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.534358 13712 raft_consensus.cc:686] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:43.534487 13712 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.536006 12451 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:43.685322 13649 raft_consensus.cc:482] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.685456 13649 raft_consensus.cc:504] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.685942 13649 leader_election.cc:288] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:43.686470 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7ea564ea504163a1e17561be9a5130" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:43.686625 12726 raft_consensus.cc:2454] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:43.686939 12528 leader_election.cc:302] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:43.687141 13649 raft_consensus.cc:2781] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.687238 13649 raft_consensus.cc:482] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.687292 13649 raft_consensus.cc:3037] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.688122 13649 raft_consensus.cc:504] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.688498 13649 leader_election.cc:288] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:43.688836 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7ea564ea504163a1e17561be9a5130" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:43.688961 12726 raft_consensus.cc:3037] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.689707 12726 raft_consensus.cc:2454] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:43.690006 12528 leader_election.cc:302] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:43.690205 13649 raft_consensus.cc:2781] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.690254 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7ea564ea504163a1e17561be9a5130" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:43.690306 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d7ea564ea504163a1e17561be9a5130" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:43.690379 13649 raft_consensus.cc:686] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Becoming Leader. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:43.690372 12988 raft_consensus.cc:2454] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:43.690517 13649 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.692221 12451 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:43.811167 13735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:43.811417 13735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:43.814276 13735 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 19:56:43.885982 12987 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.886530 13583 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.887216 12595 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.887583 13587 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.888054 12988 raft_consensus.cc:3037] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.889230 12726 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.889609 13733 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.891098 12988 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.891476 13733 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.894269 12986 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.894685 13712 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:43.894862 12594 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:43.895493 13682 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 106.906 ms
time per row: 0.0534529 ms
I20240501 19:56:43.959656 12448 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:43.964896 12575 tablet_service.cc:1467] Processing CreateTablet for tablet 32a17c2e392a49b7b181fa7a25a778a1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.965018 12705 tablet_service.cc:1467] Processing CreateTablet for tablet 32a17c2e392a49b7b181fa7a25a778a1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.965237 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a17c2e392a49b7b181fa7a25a778a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.965276 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a17c2e392a49b7b181fa7a25a778a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.965478 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 32a17c2e392a49b7b181fa7a25a778a1 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.965693 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a17c2e392a49b7b181fa7a25a778a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.967399 13803 tablet_bootstrap.cc:492] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:43.967782 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 706878734216445f8422e422228d4230 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.968027 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706878734216445f8422e422228d4230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.968277 13803 tablet_bootstrap.cc:654] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.968338 12575 tablet_service.cc:1467] Processing CreateTablet for tablet 706878734216445f8422e422228d4230 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.968467 12836 tablet_service.cc:1467] Processing CreateTablet for tablet d75bd8f836b947248ee124f456c7d721 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.968551 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706878734216445f8422e422228d4230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.968676 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75bd8f836b947248ee124f456c7d721. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.968843 13804 tablet_bootstrap.cc:492] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:43.969715 13804 tablet_bootstrap.cc:654] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.970431 12968 tablet_service.cc:1467] Processing CreateTablet for tablet d75bd8f836b947248ee124f456c7d721 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.970610 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75bd8f836b947248ee124f456c7d721. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.971144 12705 tablet_service.cc:1467] Processing CreateTablet for tablet d75bd8f836b947248ee124f456c7d721 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.971387 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75bd8f836b947248ee124f456c7d721. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.971635 13804 tablet_bootstrap.cc:492] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:43.971724 13804 ts_tablet_manager.cc:1367] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:43.972131 13804 raft_consensus.cc:348] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.972242 13804 raft_consensus.cc:374] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.972278 13804 raft_consensus.cc:729] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.972342 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 706878734216445f8422e422228d4230 (DEFAULT_TABLE table=rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:43.972545 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706878734216445f8422e422228d4230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:43.973140 13804 consensus_queue.cc:260] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.973224 13807 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:43.973524 13802 tablet_bootstrap.cc:492] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:43.974220 13807 tablet_bootstrap.cc:654] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.974388 13802 tablet_bootstrap.cc:654] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.975049 13804 ts_tablet_manager.cc:1398] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:43.975317 13803 tablet_bootstrap.cc:492] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:43.975330 13807 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:43.975333 13802 tablet_bootstrap.cc:492] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:43.975392 13803 ts_tablet_manager.cc:1367] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:43.975392 13807 ts_tablet_manager.cc:1367] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:43.975406 13802 ts_tablet_manager.cc:1367] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:43.975770 13807 raft_consensus.cc:348] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.975770 13803 raft_consensus.cc:348] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.975863 13807 raft_consensus.cc:374] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.975863 13803 raft_consensus.cc:374] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.975832 13802 raft_consensus.cc:348] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.975896 13803 raft_consensus.cc:729] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.975896 13807 raft_consensus.cc:729] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.975931 13802 raft_consensus.cc:374] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.975963 13802 raft_consensus.cc:729] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.976014 13803 consensus_queue.cc:260] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.976039 13807 consensus_queue.cc:260] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.976076 13802 consensus_queue.cc:260] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:43.976342 13802 ts_tablet_manager.cc:1398] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.976363 13807 ts_tablet_manager.cc:1398] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.976505 13807 tablet_bootstrap.cc:492] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:43.976508 13802 tablet_bootstrap.cc:492] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:43.976670 13803 ts_tablet_manager.cc:1398] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.976855 13803 tablet_bootstrap.cc:492] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:43.977288 13802 tablet_bootstrap.cc:654] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.977360 13807 tablet_bootstrap.cc:654] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.978358 13803 tablet_bootstrap.cc:654] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.978638 13807 tablet_bootstrap.cc:492] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:43.978742 13807 ts_tablet_manager.cc:1367] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:43.979171 13807 raft_consensus.cc:348] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.979283 13807 raft_consensus.cc:374] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.979315 13807 raft_consensus.cc:729] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.979468 13802 tablet_bootstrap.cc:492] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:43.979420 13807 consensus_queue.cc:260] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.979542 13802 ts_tablet_manager.cc:1367] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:43.979704 13807 ts_tablet_manager.cc:1398] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.979900 13802 raft_consensus.cc:348] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.980005 13802 raft_consensus.cc:374] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.980038 13802 raft_consensus.cc:729] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.980165 13802 consensus_queue.cc:260] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.980443 13802 ts_tablet_manager.cc:1398] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:43.980597 13802 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:43.980696 13804 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:43.981688 13802 tablet_bootstrap.cc:654] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.981581 13804 tablet_bootstrap.cc:654] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:43.983520 13804 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:43.983597 13804 ts_tablet_manager.cc:1367] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240501 19:56:43.984035 13804 raft_consensus.cc:348] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.984143 13804 raft_consensus.cc:374] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.984195 13804 raft_consensus.cc:729] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.984330 13804 consensus_queue.cc:260] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.984565 13804 ts_tablet_manager.cc:1398] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:43.985071 13803 tablet_bootstrap.cc:492] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:43.985152 13803 ts_tablet_manager.cc:1367] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20240501 19:56:43.985303 13802 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:43.985357 13802 ts_tablet_manager.cc:1367] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 19:56:43.985584 13803 raft_consensus.cc:348] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.985625 13802 raft_consensus.cc:348] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.985675 13803 raft_consensus.cc:374] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.985688 13802 raft_consensus.cc:374] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:43.985713 13803 raft_consensus.cc:729] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.985723 13802 raft_consensus.cc:729] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:43.985829 13803 consensus_queue.cc:260] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:43.985852 13802 consensus_queue.cc:260] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.986099 13803 ts_tablet_manager.cc:1398] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:43.986100 13802 ts_tablet_manager.cc:1398] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:43.987315 13773 raft_consensus.cc:482] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:43.987404 13773 raft_consensus.cc:504] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.988440 13773 leader_election.cc:288] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:43.992043 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706878734216445f8422e422228d4230" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:43.992168 12594 raft_consensus.cc:2454] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:43.992166 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706878734216445f8422e422228d4230" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:43.992296 12855 raft_consensus.cc:2454] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:43.992501 12923 leader_election.cc:302] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:43.992723 13773 raft_consensus.cc:2781] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:43.992784 13773 raft_consensus.cc:482] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:43.992810 13773 raft_consensus.cc:3037] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.993626 13773 raft_consensus.cc:504] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.993965 13773 leader_election.cc:288] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:43.994408 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706878734216445f8422e422228d4230" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:43.994431 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "706878734216445f8422e422228d4230" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:43.994508 12855 raft_consensus.cc:3037] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.994524 12594 raft_consensus.cc:3037] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:43.995347 12594 raft_consensus.cc:2454] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:43.995350 12855 raft_consensus.cc:2454] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:43.995618 12921 leader_election.cc:302] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:43.995795 13773 raft_consensus.cc:2781] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:43.995908 13773 raft_consensus.cc:686] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Becoming Leader. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:43.996171 13773 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:43.998201 12448 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68). New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:44.010847 13583 raft_consensus.cc:482] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.010941 13583 raft_consensus.cc:504] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.011343 13583 leader_election.cc:288] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:44.011754 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75bd8f836b947248ee124f456c7d721" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:44.011879 12986 raft_consensus.cc:2454] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:44.012156 12793 leader_election.cc:302] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:44.012351 13583 raft_consensus.cc:2781] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.012411 13583 raft_consensus.cc:482] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.012465 13583 raft_consensus.cc:3037] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.013269 13583 raft_consensus.cc:504] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.013578 13583 leader_election.cc:288] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Requested vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:44.013947 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75bd8f836b947248ee124f456c7d721" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:44.014063 12986 raft_consensus.cc:3037] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.014955 12986 raft_consensus.cc:2454] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:44.015235 12793 leader_election.cc:302] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:44.015297 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75bd8f836b947248ee124f456c7d721" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:44.015406 13583 raft_consensus.cc:2781] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.015388 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d75bd8f836b947248ee124f456c7d721" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:44.015460 12726 raft_consensus.cc:2454] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:44.015488 13583 raft_consensus.cc:686] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Becoming Leader. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Running, Role: LEADER
I20240501 19:56:44.015630 13583 consensus_queue.cc:237] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.017242 12448 catalog_manager.cc:5484] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: term changed from 0 to 1, leader changed from <none> to a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67). New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:44.193006 13682 raft_consensus.cc:482] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.193171 13682 raft_consensus.cc:504] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.193572 13682 leader_election.cc:288] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:44.194051 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a17c2e392a49b7b181fa7a25a778a1" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:44.194046 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a17c2e392a49b7b181fa7a25a778a1" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:44.194177 12594 raft_consensus.cc:2454] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:44.194195 12855 raft_consensus.cc:2454] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:44.194527 12659 leader_election.cc:302] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:44.194742 13682 raft_consensus.cc:2781] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.194839 13682 raft_consensus.cc:482] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.194883 13682 raft_consensus.cc:3037] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.195751 13682 raft_consensus.cc:504] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.196031 13682 leader_election.cc:288] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:44.196398 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a17c2e392a49b7b181fa7a25a778a1" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:44.196401 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a17c2e392a49b7b181fa7a25a778a1" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:44.196506 12855 raft_consensus.cc:3037] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.196581 12594 raft_consensus.cc:3037] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.197281 12855 raft_consensus.cc:2454] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:44.197377 12594 raft_consensus.cc:2454] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:44.197537 12659 leader_election.cc:302] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:44.197695 13682 raft_consensus.cc:2781] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.197767 13682 raft_consensus.cc:686] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:44.197896 13682 consensus_queue.cc:237] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.199512 12448 catalog_manager.cc:5484] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:44.306190 13821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.306407 13821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.309273 13821 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 19:56:44.383487 12594 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.383998 13773 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.384279 12725 raft_consensus.cc:3037] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.385505 12725 raft_consensus.cc:1261] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.387264 13587 consensus_queue.cc:1035] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.388463 12986 raft_consensus.cc:1261] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.388913 13587 consensus_queue.cc:1035] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.389472 12855 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.389840 13773 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.396449 12595 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.396802 12856 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.396878 13682 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.401821 13682 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 104.564 ms
time per row: 0.052282 ms
I20240501 19:56:44.455397 12451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:44.460563 12705 tablet_service.cc:1467] Processing CreateTablet for tablet 745e39ec23db4f90bb8b1e8b49a50244 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.460628 12575 tablet_service.cc:1467] Processing CreateTablet for tablet 745e39ec23db4f90bb8b1e8b49a50244 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.460883 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745e39ec23db4f90bb8b1e8b49a50244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.460883 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745e39ec23db4f90bb8b1e8b49a50244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.461031 12706 tablet_service.cc:1467] Processing CreateTablet for tablet 544caa38514643d5b81cd6ed85e0bae7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.461294 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544caa38514643d5b81cd6ed85e0bae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.461496 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 544caa38514643d5b81cd6ed85e0bae7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.461764 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544caa38514643d5b81cd6ed85e0bae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.462108 12968 tablet_service.cc:1467] Processing CreateTablet for tablet 745e39ec23db4f90bb8b1e8b49a50244 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.462368 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745e39ec23db4f90bb8b1e8b49a50244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.463261 13803 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:44.463889 13802 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:44.464107 13803 tablet_bootstrap.cc:654] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.464479 13807 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:44.464653 13802 tablet_bootstrap.cc:654] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.464669 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 851eeb0ab3364529907d9d09f2a4022e (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.464861 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851eeb0ab3364529907d9d09f2a4022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.465557 13807 tablet_bootstrap.cc:654] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.466639 13807 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:44.466735 13807 ts_tablet_manager.cc:1367] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:44.467341 13804 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:44.467381 13807 raft_consensus.cc:348] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.467491 13807 raft_consensus.cc:374] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.467535 13807 raft_consensus.cc:729] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.467646 13807 consensus_queue.cc:260] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.468017 13807 ts_tablet_manager.cc:1398] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:44.468130 13804 tablet_bootstrap.cc:654] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.468174 13807 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:44.468263 12575 tablet_service.cc:1467] Processing CreateTablet for tablet 544caa38514643d5b81cd6ed85e0bae7 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.468498 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544caa38514643d5b81cd6ed85e0bae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.468923 12574 tablet_service.cc:1467] Processing CreateTablet for tablet 851eeb0ab3364529907d9d09f2a4022e (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.469002 13807 tablet_bootstrap.cc:654] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.469364 12574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851eeb0ab3364529907d9d09f2a4022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.469967 13807 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:44.470050 13807 ts_tablet_manager.cc:1367] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:44.470487 13807 raft_consensus.cc:348] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.470584 13807 raft_consensus.cc:374] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.470613 13807 raft_consensus.cc:729] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.470713 13807 consensus_queue.cc:260] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.470964 13807 ts_tablet_manager.cc:1398] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.471557 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 851eeb0ab3364529907d9d09f2a4022e (DEFAULT_TABLE table=rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.471786 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851eeb0ab3364529907d9d09f2a4022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.473591 13803 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:44.473678 13803 ts_tablet_manager.cc:1367] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20240501 19:56:44.474000 13802 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:44.474064 13802 ts_tablet_manager.cc:1367] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20240501 19:56:44.474115 13803 raft_consensus.cc:348] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.474220 13803 raft_consensus.cc:374] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.474252 13803 raft_consensus.cc:729] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.474376 13803 consensus_queue.cc:260] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.474436 13802 raft_consensus.cc:348] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.474522 13802 raft_consensus.cc:374] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.474557 13802 raft_consensus.cc:729] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.474642 13803 ts_tablet_manager.cc:1398] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:44.474676 13802 consensus_queue.cc:260] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.474809 13803 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:44.474910 13802 ts_tablet_manager.cc:1398] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:44.475033 13802 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:44.475848 13802 tablet_bootstrap.cc:654] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.475857 13803 tablet_bootstrap.cc:654] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.476801 13803 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:44.476873 13803 ts_tablet_manager.cc:1367] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:44.477357 13803 raft_consensus.cc:348] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.477470 13803 raft_consensus.cc:374] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.477499 13803 raft_consensus.cc:729] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.477615 13803 consensus_queue.cc:260] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.477878 13803 ts_tablet_manager.cc:1398] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.478305 13804 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:44.478392 13804 ts_tablet_manager.cc:1367] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20240501 19:56:44.478752 13804 raft_consensus.cc:348] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.478842 13804 raft_consensus.cc:374] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.478878 13804 raft_consensus.cc:729] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.479014 13804 consensus_queue.cc:260] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.479260 13804 ts_tablet_manager.cc:1398] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:44.479490 13804 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:44.480360 13804 tablet_bootstrap.cc:654] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.481770 13802 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:44.481827 13804 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:44.481856 13802 ts_tablet_manager.cc:1367] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:44.481909 13804 ts_tablet_manager.cc:1367] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:44.482203 13802 raft_consensus.cc:348] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.482307 13802 raft_consensus.cc:374] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.482287 13804 raft_consensus.cc:348] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.482349 13802 raft_consensus.cc:729] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.482398 13804 raft_consensus.cc:374] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.482437 13804 raft_consensus.cc:729] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.482476 13802 consensus_queue.cc:260] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.482544 13804 consensus_queue.cc:260] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.482759 13802 ts_tablet_manager.cc:1398] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:44.482995 13804 ts_tablet_manager.cc:1398] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:44.483318 13804 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:44.484457 13804 tablet_bootstrap.cc:654] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.485533 13804 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:44.485615 13804 ts_tablet_manager.cc:1367] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:44.485952 13804 raft_consensus.cc:348] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.486044 13804 raft_consensus.cc:374] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.486080 13804 raft_consensus.cc:729] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.486202 13804 consensus_queue.cc:260] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.486418 13804 ts_tablet_manager.cc:1398] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:44.552274 13779 raft_consensus.cc:482] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.552407 13779 raft_consensus.cc:504] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.552826 13779 leader_election.cc:288] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:44.553184 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851eeb0ab3364529907d9d09f2a4022e" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:44.553210 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851eeb0ab3364529907d9d09f2a4022e" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:44.553318 12595 raft_consensus.cc:2454] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:44.553345 12856 raft_consensus.cc:2454] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:44.553601 12921 leader_election.cc:302] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:44.553778 13779 raft_consensus.cc:2781] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.553846 13779 raft_consensus.cc:482] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.553881 13779 raft_consensus.cc:3037] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.554961 13779 raft_consensus.cc:504] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.555294 13779 leader_election.cc:288] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:44.555594 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851eeb0ab3364529907d9d09f2a4022e" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:44.555588 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "851eeb0ab3364529907d9d09f2a4022e" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:44.555693 12856 raft_consensus.cc:3037] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.555701 12595 raft_consensus.cc:3037] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.556432 12595 raft_consensus.cc:2454] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:44.556581 12856 raft_consensus.cc:2454] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:44.556712 12923 leader_election.cc:302] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:44.556932 13779 raft_consensus.cc:2781] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.557031 13779 raft_consensus.cc:686] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Becoming Leader. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:44.557207 13779 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.558913 12450 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68). New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:44.590119 13779 raft_consensus.cc:482] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.590237 13779 raft_consensus.cc:504] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.590660 13779 leader_election.cc:288] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:44.591087 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:44.591233 12595 raft_consensus.cc:2454] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:44.591532 12923 leader_election.cc:302] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:44.591711 13779 raft_consensus.cc:2781] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.591765 13779 raft_consensus.cc:482] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.591810 13779 raft_consensus.cc:3037] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.592756 13779 raft_consensus.cc:504] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.593174 13779 leader_election.cc:288] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Requested vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:44.593495 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:44.593591 12595 raft_consensus.cc:3037] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.594403 12595 raft_consensus.cc:2454] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:44.594658 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:44.594699 12923 leader_election.cc:302] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:44.594710 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:44.594808 12725 raft_consensus.cc:2454] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:44.595077 13779 raft_consensus.cc:2781] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.595175 13779 raft_consensus.cc:686] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Becoming Leader. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:44.595489 13779 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.597152 12450 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68). New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:44.598011 13733 raft_consensus.cc:482] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.598093 13733 raft_consensus.cc:504] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.598445 13733 leader_election.cc:288] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:44.598802 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544caa38514643d5b81cd6ed85e0bae7" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:44.598920 12726 raft_consensus.cc:2454] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:44.599165 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544caa38514643d5b81cd6ed85e0bae7" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:44.599193 12528 leader_election.cc:302] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:44.599309 12856 raft_consensus.cc:2454] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:44.599363 13733 raft_consensus.cc:2781] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.599428 13733 raft_consensus.cc:482] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.599475 13733 raft_consensus.cc:3037] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.600198 13733 raft_consensus.cc:504] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.600486 13733 leader_election.cc:288] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:44.600812 12856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544caa38514643d5b81cd6ed85e0bae7" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:44.600873 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "544caa38514643d5b81cd6ed85e0bae7" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:44.600932 12856 raft_consensus.cc:3037] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.600953 12726 raft_consensus.cc:3037] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.601846 12726 raft_consensus.cc:2454] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:44.601846 12856 raft_consensus.cc:2454] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:44.602116 12528 leader_election.cc:302] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:44.602263 13733 raft_consensus.cc:2781] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.602398 13733 raft_consensus.cc:686] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Becoming Leader. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:44.602540 13733 consensus_queue.cc:237] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:44.603952 12450 catalog_manager.cc:5484] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:44.725149 13873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:44.725430 13873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:44.729636 13873 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 19:56:44.809463 12595 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.809597 12726 raft_consensus.cc:3037] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.810050 13780 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.810740 12856 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.810761 12726 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.811142 13779 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.811192 13773 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.816848 12856 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.818027 13733 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.818296 12595 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.819090 13780 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.828984 13786 mvcc.cc:205] Tried to move back new op lower bound from 7022974586099142656 to 7022974585067839488. Current Snapshot: MvccSnapshot[applied={T|T < 7022974586099142656}]
I20240501 19:56:44.833757 12726 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:44.836133 13733 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:44.846052 13777 mvcc.cc:205] Tried to move back new op lower bound from 7022974586095783936 to 7022974585224355840. Current Snapshot: MvccSnapshot[applied={T|T < 7022974586095783936}]
I20240501 19:56:44.851140 13770 mvcc.cc:205] Tried to move back new op lower bound from 7022974586099142656 to 7022974585067839488. Current Snapshot: MvccSnapshot[applied={T|T < 7022974586099142656 or (T in {7022974586099142656})}]
INSERT report
rows total: 2000
time total: 106.112 ms
time per row: 0.0530559 ms
I20240501 19:56:44.878634 12451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:44.884168 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 (DEFAULT_TABLE table=rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.884168 12706 tablet_service.cc:1467] Processing CreateTablet for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 (DEFAULT_TABLE table=rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.884461 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.884461 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.885318 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 (DEFAULT_TABLE table=rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.885538 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.886564 13807 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:44.886610 13803 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:44.886974 12705 tablet_service.cc:1467] Processing CreateTablet for tablet 22b272a8582d4060a6a527ac4fce9ae5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.887200 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b272a8582d4060a6a527ac4fce9ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.887414 13807 tablet_bootstrap.cc:654] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.887459 13803 tablet_bootstrap.cc:654] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.887425 12836 tablet_service.cc:1467] Processing CreateTablet for tablet ca3285abe3654723a6922672391e9423 (DEFAULT_TABLE table=rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.887655 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3285abe3654723a6922672391e9423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.888844 13803 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:44.888935 13803 ts_tablet_manager.cc:1367] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:44.889099 13802 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:44.889408 13803 raft_consensus.cc:348] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:44.889539 13803 raft_consensus.cc:374] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.889581 13803 raft_consensus.cc:729] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.889696 13803 consensus_queue.cc:260] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:44.889793 13802 tablet_bootstrap.cc:654] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.889969 13803 ts_tablet_manager.cc:1398] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:44.889943 12574 tablet_service.cc:1467] Processing CreateTablet for tablet 22b272a8582d4060a6a527ac4fce9ae5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.890136 13803 tablet_bootstrap.cc:492] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:44.890215 12574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b272a8582d4060a6a527ac4fce9ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.890501 12968 tablet_service.cc:1467] Processing CreateTablet for tablet 22b272a8582d4060a6a527ac4fce9ae5 (DEFAULT_TABLE table=rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.890699 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b272a8582d4060a6a527ac4fce9ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.890945 13803 tablet_bootstrap.cc:654] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.891935 13803 tablet_bootstrap.cc:492] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:44.892009 13803 ts_tablet_manager.cc:1367] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:44.892434 13804 tablet_bootstrap.cc:492] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:44.892436 13803 raft_consensus.cc:348] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.892524 13803 raft_consensus.cc:374] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.892553 13803 raft_consensus.cc:729] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.892657 13803 consensus_queue.cc:260] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.892911 13803 ts_tablet_manager.cc:1398] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:44.893260 13804 tablet_bootstrap.cc:654] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.893999 12967 tablet_service.cc:1467] Processing CreateTablet for tablet ca3285abe3654723a6922672391e9423 (DEFAULT_TABLE table=rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.894243 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3285abe3654723a6922672391e9423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.894757 13804 tablet_bootstrap.cc:492] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:44.894843 13804 ts_tablet_manager.cc:1367] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:44.894908 13807 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:44.894992 13807 ts_tablet_manager.cc:1367] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:44.895115 12706 tablet_service.cc:1467] Processing CreateTablet for tablet ca3285abe3654723a6922672391e9423 (DEFAULT_TABLE table=rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:44.895267 13804 raft_consensus.cc:348] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.895359 13804 raft_consensus.cc:374] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.895362 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3285abe3654723a6922672391e9423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:44.895395 13804 raft_consensus.cc:729] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.895393 13807 raft_consensus.cc:348] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:44.895490 13807 raft_consensus.cc:374] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.895525 13807 raft_consensus.cc:729] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.895534 13804 consensus_queue.cc:260] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.895634 13807 consensus_queue.cc:260] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:44.895823 13804 ts_tablet_manager.cc:1398] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.895880 13807 ts_tablet_manager.cc:1398] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.896032 13807 tablet_bootstrap.cc:492] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:44.896767 13807 tablet_bootstrap.cc:654] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.897733 13807 tablet_bootstrap.cc:492] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:44.897867 13807 ts_tablet_manager.cc:1367] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:44.898340 13807 raft_consensus.cc:348] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.898445 13807 raft_consensus.cc:374] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.898481 13807 raft_consensus.cc:729] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.898599 13807 consensus_queue.cc:260] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.898852 13807 ts_tablet_manager.cc:1398] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:44.901186 13803 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:44.901811 13807 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:44.902104 13803 tablet_bootstrap.cc:654] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.902593 13807 tablet_bootstrap.cc:654] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.903712 13802 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:44.903800 13802 ts_tablet_manager.cc:1367] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.000s
I20240501 19:56:44.904156 13802 raft_consensus.cc:348] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:44.904263 13802 raft_consensus.cc:374] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.904300 13802 raft_consensus.cc:729] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.904417 13802 consensus_queue.cc:260] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:44.904644 13802 ts_tablet_manager.cc:1398] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.904785 13802 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:44.905601 13803 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:44.905629 13802 tablet_bootstrap.cc:654] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:44.905680 13803 ts_tablet_manager.cc:1367] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240501 19:56:44.906540 13803 raft_consensus.cc:348] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.906636 13803 raft_consensus.cc:374] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.906667 13803 raft_consensus.cc:729] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.906776 13803 consensus_queue.cc:260] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.907011 13803 ts_tablet_manager.cc:1398] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.907096 13802 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:44.907174 13802 ts_tablet_manager.cc:1367] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:44.907524 13802 raft_consensus.cc:348] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.907613 13802 raft_consensus.cc:374] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.907639 13802 raft_consensus.cc:729] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.907759 13802 consensus_queue.cc:260] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.908051 13802 ts_tablet_manager.cc:1398] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:44.908447 13807 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:44.908630 13807 ts_tablet_manager.cc:1367] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:44.909153 13807 raft_consensus.cc:348] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.909261 13807 raft_consensus.cc:374] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:44.909297 13807 raft_consensus.cc:729] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:44.909405 13807 consensus_queue.cc:260] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.909693 13807 ts_tablet_manager.cc:1398] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:44.987522 13733 raft_consensus.cc:482] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:44.987654 13733 raft_consensus.cc:504] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.988130 13733 leader_election.cc:288] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:44.988539 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b272a8582d4060a6a527ac4fce9ae5" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:44.988548 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b272a8582d4060a6a527ac4fce9ae5" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:44.988668 12725 raft_consensus.cc:2454] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:44.988692 12986 raft_consensus.cc:2454] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:44.988996 12528 leader_election.cc:302] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:44.989193 13733 raft_consensus.cc:2781] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:44.989249 13733 raft_consensus.cc:482] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:44.989275 13733 raft_consensus.cc:3037] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.990170 13733 raft_consensus.cc:504] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.990518 13733 leader_election.cc:288] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Requested vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:44.990849 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b272a8582d4060a6a527ac4fce9ae5" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:44.990851 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22b272a8582d4060a6a527ac4fce9ae5" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:44.990953 12986 raft_consensus.cc:3037] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.990972 12725 raft_consensus.cc:3037] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:44.991689 12986 raft_consensus.cc:2454] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:44.991710 12725 raft_consensus.cc:2454] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:44.992007 12531 leader_election.cc:302] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:44.992194 13733 raft_consensus.cc:2781] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:44.992300 13733 raft_consensus.cc:686] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Becoming Leader. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:44.992452 13733 consensus_queue.cc:237] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:44.994081 12448 catalog_manager.cc:5484] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.042824 13712 raft_consensus.cc:482] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.042992 13712 raft_consensus.cc:504] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.043468 13712 leader_election.cc:288] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:45.043915 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800d5f43cd1b4fb6bd8bbcac262aeb80" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:45.043916 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800d5f43cd1b4fb6bd8bbcac262aeb80" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:45.044076 12986 raft_consensus.cc:2454] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:45.044075 12855 raft_consensus.cc:2454] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:45.044399 12662 leader_election.cc:302] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:45.044588 13682 raft_consensus.cc:482] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.044611 13712 raft_consensus.cc:2781] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.044672 13682 raft_consensus.cc:504] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:45.044675 13712 raft_consensus.cc:482] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.044800 13712 raft_consensus.cc:3037] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.044994 13682 leader_election.cc:288] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:45.045356 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3285abe3654723a6922672391e9423" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:45.045353 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3285abe3654723a6922672391e9423" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:45.045471 12855 raft_consensus.cc:2454] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:45.045475 12986 raft_consensus.cc:2454] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:45.045724 12659 leader_election.cc:302] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:45.045935 13682 raft_consensus.cc:2781] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.045996 13682 raft_consensus.cc:482] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.046044 13682 raft_consensus.cc:3037] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.046015 13712 raft_consensus.cc:504] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.046322 13712 leader_election.cc:288] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:45.046614 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800d5f43cd1b4fb6bd8bbcac262aeb80" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:45.046770 12855 raft_consensus.cc:3037] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.047333 13682 raft_consensus.cc:504] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:45.047695 13682 leader_election.cc:288] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:45.047755 12855 raft_consensus.cc:2454] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:45.047955 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "800d5f43cd1b4fb6bd8bbcac262aeb80" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:45.048007 12659 leader_election.cc:302] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:45.048060 12986 raft_consensus.cc:3037] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.048141 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3285abe3654723a6922672391e9423" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:45.048192 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3285abe3654723a6922672391e9423" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:45.048228 12855 raft_consensus.cc:3037] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.048293 12987 raft_consensus.cc:3037] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.048439 13682 raft_consensus.cc:2781] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.048502 13682 raft_consensus.cc:686] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:45.048632 13682 consensus_queue.cc:237] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.048961 12855 raft_consensus.cc:2454] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:45.048961 12986 raft_consensus.cc:2454] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:45.049692 12659 leader_election.cc:302] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:45.049857 13682 raft_consensus.cc:2781] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.049939 13682 raft_consensus.cc:686] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:45.050053 13682 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:45.050441 12448 catalog_manager.cc:5484] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.053695 12448 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.069780 12987 raft_consensus.cc:2454] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
W20240501 19:56:45.196244 13919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:45.196447 13919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:45.199380 13919 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 19:56:45.287070 12855 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.287568 12855 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.287669 12986 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.287874 12988 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.288203 13712 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.288383 13946 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.288686 13682 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.288792 13946 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.296247 12725 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.296655 13871 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.297566 12987 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.298373 13783 mvcc.cc:205] Tried to move back new op lower bound from 7022974588054151168 to 7022974587080355840. Current Snapshot: MvccSnapshot[applied={T|T < 7022974588054151168}]
I20240501 19:56:45.298672 13871 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 118.165 ms
time per row: 0.0590825 ms
I20240501 19:56:45.357810 12451 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:45.363020 12836 tablet_service.cc:1467] Processing CreateTablet for tablet 557994658e2c4c8bbc1eeacdab958ddd (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.363224 12574 tablet_service.cc:1467] Processing CreateTablet for tablet 557994658e2c4c8bbc1eeacdab958ddd (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.363360 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557994658e2c4c8bbc1eeacdab958ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.363512 12574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557994658e2c4c8bbc1eeacdab958ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.364022 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 955edb7c12fa4b32ac58827f131931e3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.364281 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955edb7c12fa4b32ac58827f131931e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.364334 12575 tablet_service.cc:1467] Processing CreateTablet for tablet 955edb7c12fa4b32ac58827f131931e3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.364506 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955edb7c12fa4b32ac58827f131931e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.365677 13804 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:45.365882 12706 tablet_service.cc:1467] Processing CreateTablet for tablet 557994658e2c4c8bbc1eeacdab958ddd (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.366147 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557994658e2c4c8bbc1eeacdab958ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.366613 13804 tablet_bootstrap.cc:654] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.367112 13807 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:45.367815 13807 tablet_bootstrap.cc:654] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.367883 13803 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:45.368065 12573 tablet_service.cc:1467] Processing CreateTablet for tablet 61bc22cdc6ab45529066fe6f4d14d4c7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.368278 12573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61bc22cdc6ab45529066fe6f4d14d4c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.368649 13803 tablet_bootstrap.cc:654] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.369151 13807 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:45.369225 13807 ts_tablet_manager.cc:1367] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:45.369601 13807 raft_consensus.cc:348] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.369693 13807 raft_consensus.cc:374] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.369726 13807 raft_consensus.cc:729] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.369858 13807 consensus_queue.cc:260] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.370111 13807 ts_tablet_manager.cc:1398] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.370671 13804 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:45.370757 13804 ts_tablet_manager.cc:1367] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:45.371157 13804 raft_consensus.cc:348] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:45.371271 13804 raft_consensus.cc:374] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.371320 13804 raft_consensus.cc:729] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.371265 12837 tablet_service.cc:1467] Processing CreateTablet for tablet 955edb7c12fa4b32ac58827f131931e3 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.371346 12835 tablet_service.cc:1467] Processing CreateTablet for tablet 61bc22cdc6ab45529066fe6f4d14d4c7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.371438 13804 consensus_queue.cc:260] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:45.371539 12837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955edb7c12fa4b32ac58827f131931e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.371701 13804 ts_tablet_manager.cc:1398] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.371747 12835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61bc22cdc6ab45529066fe6f4d14d4c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.371855 13804 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:45.372449 13803 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:45.372547 13803 ts_tablet_manager.cc:1367] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:45.372627 13804 tablet_bootstrap.cc:654] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.372699 12706 tablet_service.cc:1467] Processing CreateTablet for tablet 61bc22cdc6ab45529066fe6f4d14d4c7 (DEFAULT_TABLE table=rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.372936 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61bc22cdc6ab45529066fe6f4d14d4c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.373143 13803 raft_consensus.cc:348] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:45.373256 13803 raft_consensus.cc:374] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.373286 13803 raft_consensus.cc:729] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.373395 13803 consensus_queue.cc:260] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:45.373665 13803 ts_tablet_manager.cc:1398] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.374212 13804 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:45.374285 13804 ts_tablet_manager.cc:1367] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20240501 19:56:45.374703 13804 raft_consensus.cc:348] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.374807 13804 raft_consensus.cc:374] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.374837 13804 raft_consensus.cc:729] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.374975 13804 consensus_queue.cc:260] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.375242 13804 ts_tablet_manager.cc:1398] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.375389 13804 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:45.375532 13803 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:45.376312 13804 tablet_bootstrap.cc:654] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.376381 13803 tablet_bootstrap.cc:654] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.377360 13804 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:45.377431 13804 ts_tablet_manager.cc:1367] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20240501 19:56:45.377835 13804 raft_consensus.cc:348] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.377936 13804 raft_consensus.cc:374] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.377969 13804 raft_consensus.cc:729] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.378084 13804 consensus_queue.cc:260] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.378386 13804 ts_tablet_manager.cc:1398] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:45.378953 13802 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:45.379061 13803 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:45.379137 13803 ts_tablet_manager.cc:1367] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:45.379475 13803 raft_consensus.cc:348] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.379539 13803 raft_consensus.cc:374] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.379559 13803 raft_consensus.cc:729] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.379653 13803 consensus_queue.cc:260] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.379841 13802 tablet_bootstrap.cc:654] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.379850 13803 ts_tablet_manager.cc:1398] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.382206 13802 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:45.382287 13802 ts_tablet_manager.cc:1367] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:45.382644 13802 raft_consensus.cc:348] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.382751 13802 raft_consensus.cc:374] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.382782 13802 raft_consensus.cc:729] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.382885 13802 consensus_queue.cc:260] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.383105 13802 ts_tablet_manager.cc:1398] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.383299 13802 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:45.384057 13802 tablet_bootstrap.cc:654] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.385121 13802 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:45.385203 13802 ts_tablet_manager.cc:1367] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:45.385566 13802 raft_consensus.cc:348] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.385649 13802 raft_consensus.cc:374] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.385687 13802 raft_consensus.cc:729] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.385818 13802 consensus_queue.cc:260] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.386086 13802 ts_tablet_manager.cc:1398] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.386232 13802 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:45.387063 13802 tablet_bootstrap.cc:654] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.388075 13802 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:45.388132 13802 ts_tablet_manager.cc:1367] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:45.388417 13802 raft_consensus.cc:348] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:45.388474 13802 raft_consensus.cc:374] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.388499 13802 raft_consensus.cc:729] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.388628 13802 consensus_queue.cc:260] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:45.388954 13802 ts_tablet_manager.cc:1398] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.455677 13587 raft_consensus.cc:482] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.455792 13587 raft_consensus.cc:504] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:45.456153 13587 leader_election.cc:288] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:45.456470 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557994658e2c4c8bbc1eeacdab958ddd" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:45.456493 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557994658e2c4c8bbc1eeacdab958ddd" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:45.456602 12725 raft_consensus.cc:2454] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:45.456617 12594 raft_consensus.cc:2454] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:45.456915 12790 leader_election.cc:302] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:45.457104 13587 raft_consensus.cc:2781] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.457173 13587 raft_consensus.cc:482] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.457223 13587 raft_consensus.cc:3037] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.458051 13587 raft_consensus.cc:504] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:45.458381 13587 leader_election.cc:288] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Requested vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:45.458675 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557994658e2c4c8bbc1eeacdab958ddd" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:45.458729 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "557994658e2c4c8bbc1eeacdab958ddd" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:45.458786 12594 raft_consensus.cc:3037] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.458844 12725 raft_consensus.cc:3037] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.459560 13712 raft_consensus.cc:482] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.459659 13712 raft_consensus.cc:504] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.459725 12725 raft_consensus.cc:2454] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:45.459730 12594 raft_consensus.cc:2454] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:45.459982 12792 leader_election.cc:302] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:45.460014 13712 leader_election.cc:288] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:45.460317 13587 raft_consensus.cc:2781] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.460407 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61bc22cdc6ab45529066fe6f4d14d4c7" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:45.460469 13587 raft_consensus.cc:686] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Becoming Leader. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Running, Role: LEADER
I20240501 19:56:45.460536 12594 raft_consensus.cc:2454] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:45.460511 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61bc22cdc6ab45529066fe6f4d14d4c7" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:45.460621 12855 raft_consensus.cc:2454] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:45.460618 13587 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } }
I20240501 19:56:45.460955 12659 leader_election.cc:302] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:45.461351 13712 raft_consensus.cc:2781] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.461401 13712 raft_consensus.cc:482] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.461448 13712 raft_consensus.cc:3037] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.462265 13712 raft_consensus.cc:504] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.462642 13712 leader_election.cc:288] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:45.462641 12452 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: term changed from 0 to 1, leader changed from <none> to a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67). New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.462926 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61bc22cdc6ab45529066fe6f4d14d4c7" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:45.462965 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61bc22cdc6ab45529066fe6f4d14d4c7" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:45.463043 12855 raft_consensus.cc:3037] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.463055 12594 raft_consensus.cc:3037] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.463841 12594 raft_consensus.cc:2454] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:45.463907 12855 raft_consensus.cc:2454] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:45.464109 12661 leader_election.cc:302] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:45.464680 13712 raft_consensus.cc:2781] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.464834 13712 raft_consensus.cc:686] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:45.464987 13712 consensus_queue.cc:237] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.466434 12452 catalog_manager.cc:5484] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.473913 13733 raft_consensus.cc:482] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.473987 13733 raft_consensus.cc:504] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.474299 13733 leader_election.cc:288] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:45.474591 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:45.474630 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:45.474690 12855 raft_consensus.cc:2454] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:45.474773 12987 raft_consensus.cc:2454] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:45.474969 12528 leader_election.cc:302] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:45.475121 13733 raft_consensus.cc:2781] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.475176 13733 raft_consensus.cc:482] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.475219 13733 raft_consensus.cc:3037] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.475822 13733 raft_consensus.cc:504] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.476102 13733 leader_election.cc:288] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:45.476406 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:45.476451 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:45.476511 12855 raft_consensus.cc:3037] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.476542 12987 raft_consensus.cc:3037] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.477183 12855 raft_consensus.cc:2454] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:45.477301 12987 raft_consensus.cc:2454] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:45.477584 12531 leader_election.cc:302] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:45.477892 13733 raft_consensus.cc:2781] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.477965 13733 raft_consensus.cc:686] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Becoming Leader. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:45.478087 13733 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.479416 12452 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } }
W20240501 19:56:45.595854 13968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:45.596153 13968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:45.599445 13968 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
I20240501 19:56:45.677371 12594 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.677812 13587 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.679329 12725 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.680506 12855 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.680900 13682 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.681064 13587 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.681591 12593 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.682363 12987 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.682742 13733 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.685659 12856 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.686029 13733 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.686064 13712 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 109.146 ms
time per row: 0.0545731 ms
I20240501 19:56:45.751894 12449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:45.757357 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc9fe9ade8246779164d6db31b11e16 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.757542 12706 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc9fe9ade8246779164d6db31b11e16 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.757684 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9fe9ade8246779164d6db31b11e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.757814 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9fe9ade8246779164d6db31b11e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.759560 13807 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:45.759821 12575 tablet_service.cc:1467] Processing CreateTablet for tablet ce6add2f19a649b3a12497951047705c (DEFAULT_TABLE table=rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.759976 12573 tablet_service.cc:1467] Processing CreateTablet for tablet 8bc9fe9ade8246779164d6db31b11e16 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.760076 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6add2f19a649b3a12497951047705c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.760185 12573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9fe9ade8246779164d6db31b11e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.760443 13807 tablet_bootstrap.cc:654] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.761765 13804 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:45.761905 13803 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:45.761993 13807 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:45.762068 13807 ts_tablet_manager.cc:1367] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:45.762576 13804 tablet_bootstrap.cc:654] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.762552 13807 raft_consensus.cc:348] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.762673 13807 raft_consensus.cc:374] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.762710 13803 tablet_bootstrap.cc:654] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.762724 13807 raft_consensus.cc:729] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.762851 13807 consensus_queue.cc:260] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.763185 13807 ts_tablet_manager.cc:1398] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.763396 12968 tablet_service.cc:1467] Processing CreateTablet for tablet c7ddb6d79d384d39a709b5824f936d53 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.763619 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ddb6d79d384d39a709b5824f936d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.763967 13803 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:45.764035 13803 ts_tablet_manager.cc:1367] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:45.764302 12836 tablet_service.cc:1467] Processing CreateTablet for tablet ce6add2f19a649b3a12497951047705c (DEFAULT_TABLE table=rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.764434 13803 raft_consensus.cc:348] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.764532 13803 raft_consensus.cc:374] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.764572 13803 raft_consensus.cc:729] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.764592 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6add2f19a649b3a12497951047705c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.764694 13803 consensus_queue.cc:260] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.764884 12706 tablet_service.cc:1467] Processing CreateTablet for tablet c7ddb6d79d384d39a709b5824f936d53 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.765002 13803 ts_tablet_manager.cc:1398] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.765116 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ddb6d79d384d39a709b5824f936d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.765658 12835 tablet_service.cc:1467] Processing CreateTablet for tablet c7ddb6d79d384d39a709b5824f936d53 (DEFAULT_TABLE table=rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.765836 12835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ddb6d79d384d39a709b5824f936d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.766713 13802 tablet_bootstrap.cc:492] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:45.767596 13802 tablet_bootstrap.cc:654] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.767683 13803 tablet_bootstrap.cc:492] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:45.768414 13803 tablet_bootstrap.cc:654] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.768697 13802 tablet_bootstrap.cc:492] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:45.768790 13802 ts_tablet_manager.cc:1367] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:45.769232 13802 raft_consensus.cc:348] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.769337 13802 raft_consensus.cc:374] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.769369 13802 raft_consensus.cc:729] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.769490 13802 consensus_queue.cc:260] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.769555 13804 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:45.769625 13804 ts_tablet_manager.cc:1367] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
I20240501 19:56:45.769820 13802 ts_tablet_manager.cc:1398] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.769990 13802 tablet_bootstrap.cc:492] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:45.770074 13804 raft_consensus.cc:348] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.770174 13804 raft_consensus.cc:374] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.770206 13804 raft_consensus.cc:729] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.770318 13804 consensus_queue.cc:260] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.770576 13804 ts_tablet_manager.cc:1398] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:45.770748 13804 tablet_bootstrap.cc:492] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:45.770834 13802 tablet_bootstrap.cc:654] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.771255 13807 tablet_bootstrap.cc:492] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:45.771832 13804 tablet_bootstrap.cc:654] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.772002 13803 tablet_bootstrap.cc:492] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:45.772081 13803 ts_tablet_manager.cc:1367] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:45.772183 13807 tablet_bootstrap.cc:654] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.772492 13803 raft_consensus.cc:348] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.772603 13803 raft_consensus.cc:374] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.772634 13803 raft_consensus.cc:729] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.772759 13803 consensus_queue.cc:260] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.773032 13803 ts_tablet_manager.cc:1398] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.773408 13807 tablet_bootstrap.cc:492] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:45.773509 13807 ts_tablet_manager.cc:1367] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:45.774091 13807 raft_consensus.cc:348] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.774195 13807 raft_consensus.cc:374] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.774230 13807 raft_consensus.cc:729] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.774334 13807 consensus_queue.cc:260] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.774607 13807 ts_tablet_manager.cc:1398] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:45.775364 12967 tablet_service.cc:1467] Processing CreateTablet for tablet ce6add2f19a649b3a12497951047705c (DEFAULT_TABLE table=rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:45.775596 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6add2f19a649b3a12497951047705c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:45.777851 13804 tablet_bootstrap.cc:492] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:45.777937 13804 ts_tablet_manager.cc:1367] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240501 19:56:45.777973 13802 tablet_bootstrap.cc:492] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:45.778057 13802 ts_tablet_manager.cc:1367] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:45.778153 13807 tablet_bootstrap.cc:492] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:45.778366 13804 raft_consensus.cc:348] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.778450 13804 raft_consensus.cc:374] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.778482 13804 raft_consensus.cc:729] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.778478 13802 raft_consensus.cc:348] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.778585 13802 raft_consensus.cc:374] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.778625 13802 raft_consensus.cc:729] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.778601 13804 consensus_queue.cc:260] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.778756 13802 consensus_queue.cc:260] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.778848 13804 ts_tablet_manager.cc:1398] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:45.779016 13807 tablet_bootstrap.cc:654] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:45.779049 13802 ts_tablet_manager.cc:1398] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.780617 13807 tablet_bootstrap.cc:492] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:45.780701 13807 ts_tablet_manager.cc:1367] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240501 19:56:45.781189 13807 raft_consensus.cc:348] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.781283 13807 raft_consensus.cc:374] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:45.781319 13807 raft_consensus.cc:729] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:45.781455 13807 consensus_queue.cc:260] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.781788 13807 ts_tablet_manager.cc:1398] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:45.785365 13780 raft_consensus.cc:482] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.785439 13780 raft_consensus.cc:504] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.785770 13780 leader_election.cc:288] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:45.786134 12854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ddb6d79d384d39a709b5824f936d53" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:45.786096 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ddb6d79d384d39a709b5824f936d53" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:45.786270 12725 raft_consensus.cc:2454] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:45.786270 12854 raft_consensus.cc:2454] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:45.786510 12921 leader_election.cc:302] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:45.786779 13780 raft_consensus.cc:2781] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.786839 13780 raft_consensus.cc:482] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.786893 13780 raft_consensus.cc:3037] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.787596 13780 raft_consensus.cc:504] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.787710 13773 raft_consensus.cc:482] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.787767 13773 raft_consensus.cc:504] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.787923 13780 leader_election.cc:288] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Requested vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:45.788019 13773 leader_election.cc:288] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:45.788177 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ddb6d79d384d39a709b5824f936d53" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:45.788321 12725 raft_consensus.cc:3037] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.788373 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6add2f19a649b3a12497951047705c" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:45.788379 12594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6add2f19a649b3a12497951047705c" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:45.788496 12855 raft_consensus.cc:2454] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:45.788519 12594 raft_consensus.cc:2454] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:45.788695 12854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ddb6d79d384d39a709b5824f936d53" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:45.788769 12921 leader_election.cc:302] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:45.788801 12854 raft_consensus.cc:3037] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.788977 13773 raft_consensus.cc:2781] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.789023 13773 raft_consensus.cc:482] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.789083 13773 raft_consensus.cc:3037] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.789244 12725 raft_consensus.cc:2454] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:45.789551 12921 leader_election.cc:302] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:45.789775 12854 raft_consensus.cc:2454] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:45.789815 14018 raft_consensus.cc:2781] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.789882 13773 raft_consensus.cc:504] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.790009 14018 raft_consensus.cc:686] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Becoming Leader. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:45.790150 14018 consensus_queue.cc:237] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.790232 13773 leader_election.cc:288] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Requested vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:45.790491 12593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6add2f19a649b3a12497951047705c" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:45.790606 12593 raft_consensus.cc:3037] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.791636 12593 raft_consensus.cc:2454] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:45.791958 12923 leader_election.cc:302] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:45.792152 14018 raft_consensus.cc:2781] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.792297 14018 raft_consensus.cc:686] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Becoming Leader. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:45.792354 12854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6add2f19a649b3a12497951047705c" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:45.792418 14018 consensus_queue.cc:237] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:45.792474 12854 raft_consensus.cc:3037] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.792474 12452 catalog_manager.cc:5484] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68). New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.793155 13682 raft_consensus.cc:482] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:45.793259 12854 raft_consensus.cc:2454] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:45.793246 13682 raft_consensus.cc:504] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.793625 13682 leader_election.cc:288] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:45.793936 12593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9fe9ade8246779164d6db31b11e16" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:45.794018 12593 raft_consensus.cc:2454] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:45.794137 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9fe9ade8246779164d6db31b11e16" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:45.794265 12987 raft_consensus.cc:2454] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:45.794281 12661 leader_election.cc:302] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:45.794755 13682 raft_consensus.cc:2781] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:45.794832 13682 raft_consensus.cc:482] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:45.794874 13682 raft_consensus.cc:3037] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.795701 13682 raft_consensus.cc:504] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.796033 13682 leader_election.cc:288] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:45.796185 12452 catalog_manager.cc:5484] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68). New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:45.796303 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9fe9ade8246779164d6db31b11e16" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:45.796361 12593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9fe9ade8246779164d6db31b11e16" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:45.796411 12987 raft_consensus.cc:3037] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.796463 12593 raft_consensus.cc:3037] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:45.797093 12987 raft_consensus.cc:2454] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:45.797363 12593 raft_consensus.cc:2454] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:45.797387 12662 leader_election.cc:302] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:45.797546 13682 raft_consensus.cc:2781] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:45.797670 13682 raft_consensus.cc:686] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:45.797801 13682 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:45.799352 12452 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:45.911420 14023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:45.911680 14023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:45.914975 14023 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 19:56:45.989744 12725 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.989997 12854 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.990451 14019 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.990792 14019 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.998078 12593 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.998203 12594 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:45.998416 13780 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:45.998481 12854 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:46.000986 12987 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:46.001511 14019 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.005131 13948 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.007434 13682 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 105.501 ms
time per row: 0.0527505 ms
I20240501 19:56:46.062207 12444 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:46.068095 12575 tablet_service.cc:1467] Processing CreateTablet for tablet f08535cb2f46464b9e510a45fee67b4f (DEFAULT_TABLE table=rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.068434 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08535cb2f46464b9e510a45fee67b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.068473 12573 tablet_service.cc:1467] Processing CreateTablet for tablet d940b16d07d54945abe4a10a11dd6a2a (DEFAULT_TABLE table=rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.068686 12573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940b16d07d54945abe4a10a11dd6a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.068696 12706 tablet_service.cc:1467] Processing CreateTablet for tablet f08535cb2f46464b9e510a45fee67b4f (DEFAULT_TABLE table=rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.068994 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08535cb2f46464b9e510a45fee67b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.069276 12967 tablet_service.cc:1467] Processing CreateTablet for tablet f08535cb2f46464b9e510a45fee67b4f (DEFAULT_TABLE table=rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.069538 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08535cb2f46464b9e510a45fee67b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.070982 12705 tablet_service.cc:1467] Processing CreateTablet for tablet d940b16d07d54945abe4a10a11dd6a2a (DEFAULT_TABLE table=rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.071166 12574 tablet_service.cc:1467] Processing CreateTablet for tablet cf6948589837482a9f580731e3b861b8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.071208 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940b16d07d54945abe4a10a11dd6a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.071386 12574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6948589837482a9f580731e3b861b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.071637 13807 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:46.072571 13807 tablet_bootstrap.cc:654] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.073197 13803 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:46.073268 13804 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:46.073362 12968 tablet_service.cc:1467] Processing CreateTablet for tablet cf6948589837482a9f580731e3b861b8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.073526 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6948589837482a9f580731e3b861b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.074067 13804 tablet_bootstrap.cc:654] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.074324 13803 tablet_bootstrap.cc:654] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.075115 13804 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:46.075196 13804 ts_tablet_manager.cc:1367] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:46.075203 12835 tablet_service.cc:1467] Processing CreateTablet for tablet d940b16d07d54945abe4a10a11dd6a2a (DEFAULT_TABLE table=rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.075469 12835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940b16d07d54945abe4a10a11dd6a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.075603 13803 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:46.075618 13804 raft_consensus.cc:348] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:46.075697 13803 ts_tablet_manager.cc:1367] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:46.075731 13804 raft_consensus.cc:374] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.075765 13804 raft_consensus.cc:729] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.075881 13804 consensus_queue.cc:260] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:46.076121 13804 ts_tablet_manager.cc:1398] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:46.076082 13803 raft_consensus.cc:348] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.076211 13803 raft_consensus.cc:374] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.076246 13803 raft_consensus.cc:729] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.076269 13804 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:46.076359 13803 consensus_queue.cc:260] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.076673 13803 ts_tablet_manager.cc:1398] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.076830 13803 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:46.077654 13802 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:46.077778 13803 tablet_bootstrap.cc:654] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.077785 13804 tablet_bootstrap.cc:654] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.078565 13802 tablet_bootstrap.cc:654] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.078687 13807 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:46.078773 13807 ts_tablet_manager.cc:1367] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20240501 19:56:46.079182 13807 raft_consensus.cc:348] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.079285 13807 raft_consensus.cc:374] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.079315 13807 raft_consensus.cc:729] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.079411 13807 consensus_queue.cc:260] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.079658 13807 ts_tablet_manager.cc:1398] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:46.079851 13807 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:46.075198 12836 tablet_service.cc:1467] Processing CreateTablet for tablet cf6948589837482a9f580731e3b861b8 (DEFAULT_TABLE table=rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.080125 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6948589837482a9f580731e3b861b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.080655 13807 tablet_bootstrap.cc:654] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.081557 13802 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:46.081645 13802 ts_tablet_manager.cc:1367] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20240501 19:56:46.081645 13807 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:46.081720 13807 ts_tablet_manager.cc:1367] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.082087 13802 raft_consensus.cc:348] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:46.082120 13807 raft_consensus.cc:348] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.082182 13802 raft_consensus.cc:374] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.082208 13807 raft_consensus.cc:374] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.082221 13802 raft_consensus.cc:729] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.082252 13807 raft_consensus.cc:729] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.082345 13802 consensus_queue.cc:260] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:46.082378 13807 consensus_queue.cc:260] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.082631 13802 ts_tablet_manager.cc:1398] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.082646 13807 ts_tablet_manager.cc:1398] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:46.082790 13802 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:46.083717 13802 tablet_bootstrap.cc:654] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.084774 13802 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:46.084864 13802 ts_tablet_manager.cc:1367] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.085384 13802 raft_consensus.cc:348] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.085481 13802 raft_consensus.cc:374] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.085515 13802 raft_consensus.cc:729] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.085603 13803 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:46.085620 13802 consensus_queue.cc:260] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.085681 13803 ts_tablet_manager.cc:1367] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20240501 19:56:46.085718 13804 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:46.085796 13804 ts_tablet_manager.cc:1367] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.001s
I20240501 19:56:46.085959 13802 ts_tablet_manager.cc:1398] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.086081 13803 raft_consensus.cc:348] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:46.086186 13803 raft_consensus.cc:374] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.086217 13803 raft_consensus.cc:729] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.086228 13804 raft_consensus.cc:348] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.086323 13804 raft_consensus.cc:374] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.086319 13803 consensus_queue.cc:260] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:46.086365 13804 raft_consensus.cc:729] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.086467 13804 consensus_queue.cc:260] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.086732 13804 ts_tablet_manager.cc:1398] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:46.087347 13804 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:46.087690 13803 ts_tablet_manager.cc:1398] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.088239 13804 tablet_bootstrap.cc:654] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.089552 13804 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:46.089629 13804 ts_tablet_manager.cc:1367] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20240501 19:56:46.089985 13804 raft_consensus.cc:348] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.090090 13804 raft_consensus.cc:374] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.090123 13804 raft_consensus.cc:729] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.090390 13804 consensus_queue.cc:260] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.090770 13804 ts_tablet_manager.cc:1398] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 19:56:46.128212 13773 raft_consensus.cc:482] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.128312 13773 raft_consensus.cc:504] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.128645 13773 leader_election.cc:288] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:46.128986 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08535cb2f46464b9e510a45fee67b4f" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:46.128986 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08535cb2f46464b9e510a45fee67b4f" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:46.129144 12725 raft_consensus.cc:2454] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:46.129144 12595 raft_consensus.cc:2454] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 0.
I20240501 19:56:46.129505 12921 leader_election.cc:302] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:46.129681 13773 raft_consensus.cc:2781] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.129738 13773 raft_consensus.cc:482] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.129776 13773 raft_consensus.cc:3037] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.130510 13773 raft_consensus.cc:504] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.130798 13773 leader_election.cc:288] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Requested vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:46.131083 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08535cb2f46464b9e510a45fee67b4f" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:46.131109 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f08535cb2f46464b9e510a45fee67b4f" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:46.131171 12595 raft_consensus.cc:3037] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.131265 12725 raft_consensus.cc:3037] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.131999 12595 raft_consensus.cc:2454] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:46.132043 12725 raft_consensus.cc:2454] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 1.
I20240501 19:56:46.132292 12923 leader_election.cc:302] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:46.132454 13773 raft_consensus.cc:2781] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.132545 13773 raft_consensus.cc:686] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Becoming Leader. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:46.132706 13773 consensus_queue.cc:237] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.134627 12449 catalog_manager.cc:5484] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68). New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:46.181777 14008 raft_consensus.cc:482] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.181877 14008 raft_consensus.cc:504] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:46.182173 14008 leader_election.cc:288] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:46.182543 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940b16d07d54945abe4a10a11dd6a2a" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:46.182586 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940b16d07d54945abe4a10a11dd6a2a" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:46.182704 12855 raft_consensus.cc:2454] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:46.182715 12725 raft_consensus.cc:2454] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:46.183014 12528 leader_election.cc:302] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:46.183203 14008 raft_consensus.cc:2781] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.183274 14008 raft_consensus.cc:482] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.183315 14008 raft_consensus.cc:3037] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.184159 14008 raft_consensus.cc:504] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:46.184415 14008 leader_election.cc:288] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Requested vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:46.184784 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940b16d07d54945abe4a10a11dd6a2a" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:46.184780 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d940b16d07d54945abe4a10a11dd6a2a" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:46.184882 12855 raft_consensus.cc:3037] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.184908 12725 raft_consensus.cc:3037] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.185598 12855 raft_consensus.cc:2454] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:46.185703 12725 raft_consensus.cc:2454] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:46.185871 12528 leader_election.cc:302] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:46.186023 14008 raft_consensus.cc:2781] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.186112 14008 raft_consensus.cc:686] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Becoming Leader. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:46.186254 14008 consensus_queue.cc:237] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }
I20240501 19:56:46.187968 12449 catalog_manager.cc:5484] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:46.367866 14008 raft_consensus.cc:482] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.367992 14008 raft_consensus.cc:504] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.368471 14008 leader_election.cc:288] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:46.368887 12854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
I20240501 19:56:46.368888 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:46.369012 12854 raft_consensus.cc:2454] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:46.369030 12987 raft_consensus.cc:2454] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 0.
I20240501 19:56:46.369341 12528 leader_election.cc:302] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:46.369500 14008 raft_consensus.cc:2781] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.369560 14008 raft_consensus.cc:482] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.369606 14008 raft_consensus.cc:3037] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.370393 14008 raft_consensus.cc:504] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.370716 14008 leader_election.cc:288] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:46.371052 12854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:46.371052 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:46.371153 12854 raft_consensus.cc:3037] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.371191 12987 raft_consensus.cc:3037] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.372023 12854 raft_consensus.cc:2454] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:46.372171 12987 raft_consensus.cc:2454] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 1.
I20240501 19:56:46.372424 12528 leader_election.cc:302] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:46.372617 14008 raft_consensus.cc:2781] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.372728 14008 raft_consensus.cc:686] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Becoming Leader. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:46.372867 14008 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.374486 12449 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: term changed from 0 to 1, leader changed from <none> to ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65). New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:46.551085 14069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:46.551366 14069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:46.554582 14069 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 19:56:46.608983 14008 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.615717 13733 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.642107 12725 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:46.642582 13780 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.642823 12987 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:46.643112 12854 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:46.643293 12595 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:46.644094 13733 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.644368 13733 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:46.651940 13780 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 119.635 ms
time per row: 0.0598175 ms
I20240501 19:56:46.715929 12449 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46642:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 19:56:46.721835 12705 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf64d9f3ce94db28b032cdcd795a255 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.722169 12705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf64d9f3ce94db28b032cdcd795a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.722580 12706 tablet_service.cc:1467] Processing CreateTablet for tablet 04ccc8addc55441b9ab9d8265b90b03b (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.722813 12706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ccc8addc55441b9ab9d8265b90b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.722817 12704 tablet_service.cc:1467] Processing CreateTablet for tablet ae2885f7933f462c96db9a99a4eb2090 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.723091 12704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2885f7933f462c96db9a99a4eb2090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.724779 12836 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf64d9f3ce94db28b032cdcd795a255 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.724972 12966 tablet_service.cc:1467] Processing CreateTablet for tablet ae2885f7933f462c96db9a99a4eb2090 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.725294 12966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2885f7933f462c96db9a99a4eb2090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.725278 12835 tablet_service.cc:1467] Processing CreateTablet for tablet 04ccc8addc55441b9ab9d8265b90b03b (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.725356 14130 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:46.725739 12968 tablet_service.cc:1467] Processing CreateTablet for tablet 0bf64d9f3ce94db28b032cdcd795a255 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.725966 12968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf64d9f3ce94db28b032cdcd795a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.726373 14130 tablet_bootstrap.cc:654] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.727520 14132 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:46.727583 14130 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:46.727671 14130 ts_tablet_manager.cc:1367] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.728058 14130 raft_consensus.cc:348] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.728216 14130 raft_consensus.cc:374] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.728191 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 04ccc8addc55441b9ab9d8265b90b03b (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.728252 14130 raft_consensus.cc:729] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.728403 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ccc8addc55441b9ab9d8265b90b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.728360 14130 consensus_queue.cc:260] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.729404 14130 ts_tablet_manager.cc:1398] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.729554 14130 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:46.729535 14132 tablet_bootstrap.cc:654] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.731361 12836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf64d9f3ce94db28b032cdcd795a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.731604 12835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ccc8addc55441b9ab9d8265b90b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.731846 14130 tablet_bootstrap.cc:654] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.732169 12575 tablet_service.cc:1467] Processing CreateTablet for tablet ae2885f7933f462c96db9a99a4eb2090 (DEFAULT_TABLE table=rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 19:56:46.732430 12575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2885f7933f462c96db9a99a4eb2090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:46.733757 14130 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:46.733843 14130 ts_tablet_manager.cc:1367] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:46.734244 14130 raft_consensus.cc:348] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.734364 14130 raft_consensus.cc:374] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.734402 14130 raft_consensus.cc:729] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.734529 14130 consensus_queue.cc:260] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.734795 14130 ts_tablet_manager.cc:1398] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.734930 14130 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:46.735344 14135 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:46.735705 14130 tablet_bootstrap.cc:654] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.736196 14135 tablet_bootstrap.cc:654] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.737149 14136 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 19:56:46.737238 14135 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:46.737324 14135 ts_tablet_manager.cc:1367] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:46.737735 14135 raft_consensus.cc:348] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.737836 14135 raft_consensus.cc:374] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.737869 14135 raft_consensus.cc:729] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.737947 14136 tablet_bootstrap.cc:654] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.738194 14135 consensus_queue.cc:260] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.738495 14135 ts_tablet_manager.cc:1398] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.738730 14135 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:46.739696 14135 tablet_bootstrap.cc:654] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.740677 14135 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e: No bootstrap required, opened a new log
I20240501 19:56:46.740769 14135 ts_tablet_manager.cc:1367] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:46.741298 14135 raft_consensus.cc:348] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.741387 14135 raft_consensus.cc:374] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.741423 14135 raft_consensus.cc:729] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.741525 14135 consensus_queue.cc:260] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.741829 14135 ts_tablet_manager.cc:1398] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.745867 14130 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411: No bootstrap required, opened a new log
I20240501 19:56:46.745950 14130 ts_tablet_manager.cc:1367] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20240501 19:56:46.745982 14132 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:46.746090 14132 ts_tablet_manager.cc:1367] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20240501 19:56:46.746289 14130 raft_consensus.cc:348] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.746400 14130 raft_consensus.cc:374] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.746428 14130 raft_consensus.cc:729] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.746459 14132 raft_consensus.cc:348] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.746523 14130 consensus_queue.cc:260] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.746567 14132 raft_consensus.cc:374] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.746605 14132 raft_consensus.cc:729] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.746771 14130 ts_tablet_manager.cc:1398] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:46.746747 14132 consensus_queue.cc:260] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.747004 14132 ts_tablet_manager.cc:1398] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:46.747207 14132 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:46.748039 14132 tablet_bootstrap.cc:654] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.753695 14136 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415: No bootstrap required, opened a new log
I20240501 19:56:46.753753 14132 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:46.753785 14136 ts_tablet_manager.cc:1367] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240501 19:56:46.753832 14132 ts_tablet_manager.cc:1367] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240501 19:56:46.754137 14136 raft_consensus.cc:348] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.754243 14136 raft_consensus.cc:374] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.754274 14136 raft_consensus.cc:729] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.754215 14132 raft_consensus.cc:348] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.754318 14132 raft_consensus.cc:374] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.754359 14132 raft_consensus.cc:729] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.754449 14136 consensus_queue.cc:260] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.754487 14132 consensus_queue.cc:260] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.754751 14132 ts_tablet_manager.cc:1398] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:46.754812 14136 ts_tablet_manager.cc:1398] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 19:56:46.754905 14132 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:46.755760 14132 tablet_bootstrap.cc:654] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Neither blocks nor log segments found. Creating new log.
I20240501 19:56:46.761648 14132 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: No bootstrap required, opened a new log
I20240501 19:56:46.761744 14132 ts_tablet_manager.cc:1367] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20240501 19:56:46.762073 14132 raft_consensus.cc:348] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.762156 14132 raft_consensus.cc:374] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 19:56:46.762192 14132 raft_consensus.cc:729] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: FOLLOWER
I20240501 19:56:46.762318 14132 consensus_queue.cc:260] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.762559 14132 ts_tablet_manager.cc:1398] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:46.792100 14141 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.792196 14141 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.792495 14141 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:46.792830 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ccc8addc55441b9ab9d8265b90b03b" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:46.792840 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ccc8addc55441b9ab9d8265b90b03b" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:46.792935 12987 raft_consensus.cc:2454] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:46.792966 12726 raft_consensus.cc:2454] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:46.793246 12793 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:46.793419 14141 raft_consensus.cc:2781] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.793478 14141 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.793530 14141 raft_consensus.cc:3037] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.794317 14141 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.794657 14141 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:46.794921 14141 raft_consensus.cc:482] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.794926 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ccc8addc55441b9ab9d8265b90b03b" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:46.794950 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ccc8addc55441b9ab9d8265b90b03b" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:46.795006 14141 raft_consensus.cc:504] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.795043 12987 raft_consensus.cc:3037] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.795054 12726 raft_consensus.cc:3037] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.795367 14141 leader_election.cc:288] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:46.795655 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf64d9f3ce94db28b032cdcd795a255" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 19:56:46.795789 12725 raft_consensus.cc:2454] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:46.796074 12987 raft_consensus.cc:2454] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:46.796077 12790 leader_election.cc:302] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:46.796108 12726 raft_consensus.cc:2454] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:46.796247 14141 raft_consensus.cc:2781] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.796300 14141 raft_consensus.cc:482] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.796339 14141 raft_consensus.cc:3037] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.796329 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf64d9f3ce94db28b032cdcd795a255" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:46.796360 12793 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:46.796425 12988 raft_consensus.cc:2454] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 0.
I20240501 19:56:46.796564 13587 raft_consensus.cc:2781] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.796761 13587 raft_consensus.cc:686] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Becoming Leader. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Running, Role: LEADER
I20240501 19:56:46.796892 13587 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.797235 14141 raft_consensus.cc:504] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.797554 14141 leader_election.cc:288] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Requested vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:46.797782 12726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf64d9f3ce94db28b032cdcd795a255" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:46.797878 12726 raft_consensus.cc:3037] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.797994 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bf64d9f3ce94db28b032cdcd795a255" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:46.798081 12988 raft_consensus.cc:3037] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.798676 12726 raft_consensus.cc:2454] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:46.798692 12449 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: term changed from 0 to 1, leader changed from <none> to a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67). New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:46.798941 12988 raft_consensus.cc:2454] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 1.
I20240501 19:56:46.798962 12790 leader_election.cc:302] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:46.799232 14141 raft_consensus.cc:2781] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.799341 14141 raft_consensus.cc:686] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Becoming Leader. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Running, Role: LEADER
I20240501 19:56:46.799477 14141 consensus_queue.cc:237] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }
I20240501 19:56:46.801993 12449 catalog_manager.cc:5484] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: term changed from 0 to 1, leader changed from <none> to a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67). New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } }
I20240501 19:56:46.968031 13948 raft_consensus.cc:482] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 19:56:46.968149 13948 raft_consensus.cc:504] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.968497 13948 leader_election.cc:288] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:46.968873 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2885f7933f462c96db9a99a4eb2090" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" is_pre_election: true
I20240501 19:56:46.968878 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2885f7933f462c96db9a99a4eb2090" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 19:56:46.969012 12595 raft_consensus.cc:2454] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:46.969022 12988 raft_consensus.cc:2454] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 0.
I20240501 19:56:46.969341 12662 leader_election.cc:302] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:46.969503 13948 raft_consensus.cc:2781] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 19:56:46.969570 13948 raft_consensus.cc:482] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 19:56:46.969602 13948 raft_consensus.cc:3037] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.970412 13948 raft_consensus.cc:504] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.970691 13948 leader_election.cc:288] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Requested vote from peers ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:46.971009 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2885f7933f462c96db9a99a4eb2090" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:46.971012 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae2885f7933f462c96db9a99a4eb2090" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:46.971097 12595 raft_consensus.cc:3037] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.971104 12988 raft_consensus.cc:3037] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 0 FOLLOWER]: Advancing to term 1
I20240501 19:56:46.971810 12595 raft_consensus.cc:2454] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:46.971838 12988 raft_consensus.cc:2454] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 1.
I20240501 19:56:46.972056 12661 leader_election.cc:302] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:46.972210 13948 raft_consensus.cc:2781] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 19:56:46.972302 13948 raft_consensus.cc:686] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:46.972445 13948 consensus_queue.cc:237] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } }
I20240501 19:56:46.974133 12449 catalog_manager.cc:5484] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:47.128844 14148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:47.129127 14148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:47.132161 14148 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 19:56:47.201272 14141 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.205581 14144 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.215710 13587 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.219043 12726 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:47.219430 14141 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.219604 12988 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:47.219967 13948 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 19:56:47.222314 12595 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 19:56:47.223148 13948 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 110.001 ms
time per row: 0.0550004 ms
I20240501 19:56:47.285634 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.69:34719
--local_ip_for_outbound_sockets=127.5.139.69
--tserver_master_addrs=127.5.139.126:35403
--webserver_port=40195
--webserver_interface=127.5.139.69
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 19:56:47.395915 14204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:47.396231 14204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:47.396306 14204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:47.400401 14204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:47.400480 14204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:47.400565 14204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.69
I20240501 19:56:47.405328 14204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.69:34719
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.139.69
--webserver_port=40195
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.14204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.69
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:47.406234 14204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:47.407487 14204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:47.416282 14209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.416843 14212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.417374 14210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:47.418005 14204 server_base.cc:1034] running on GCE node
I20240501 19:56:47.418361 14204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:47.418819 14204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:47.419955 14204 hybrid_clock.cc:648] HybridClock initialized: now 1714593407419934 us; error 34 us; skew 500 ppm
I20240501 19:56:47.421705 14204 webserver.cc:460] Webserver started at http://127.5.139.69:40195/ using document root <none> and password file <none>
I20240501 19:56:47.422219 14204 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:47.422287 14204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:47.425374 14204 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 19:56:47.427198 14218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.428304 14204 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:47.428408 14204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9069ec6fcdb248479037f2d37e0e851d"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:47.428787 14204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:47.442498 14204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:47.443030 14204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:47.443195 14204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:47.443668 14204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:47.444496 14204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:47.444542 14204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.444581 14204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:47.444614 14204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.453153 14204 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.69:34719
I20240501 19:56:47.453209 14331 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.69:34719 every 8 connection(s)
I20240501 19:56:47.454056 14204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 19:56:47.460980 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 14204
I20240501 19:56:47.464049 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.70:33933
--local_ip_for_outbound_sockets=127.5.139.70
--tserver_master_addrs=127.5.139.126:35403
--webserver_port=39599
--webserver_interface=127.5.139.70
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:47.466192 14332 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:47.466413 14332 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:47.467018 14332 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:47.468176 12449 ts_manager.cc:194] Re-registered known tserver with Master: 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719)
I20240501 19:56:47.468883 12449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.69:58091
W20240501 19:56:47.572772 14335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:47.572957 14335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:47.573000 14335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:47.576529 14335 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:47.576591 14335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:47.576665 14335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.70
I20240501 19:56:47.580106 14335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.70:33933
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.5.139.70
--webserver_port=39599
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.14335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.70
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:47.580888 14335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:47.581902 14335 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:47.590133 14341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.590761 14343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.591251 14340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:47.591898 14335 server_base.cc:1034] running on GCE node
I20240501 19:56:47.592267 14335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:47.592756 14335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:47.593895 14335 hybrid_clock.cc:648] HybridClock initialized: now 1714593407593871 us; error 37 us; skew 500 ppm
I20240501 19:56:47.595695 14335 webserver.cc:460] Webserver started at http://127.5.139.70:39599/ using document root <none> and password file <none>
I20240501 19:56:47.596297 14335 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:47.596382 14335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:47.599488 14335 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:47.601229 14349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.602105 14335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:47.602208 14335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "2ca06dad25a645bca71b09741798194a"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:47.602586 14335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:47.615444 14335 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:47.615913 14335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:47.616055 14335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:47.616528 14335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:47.617326 14335 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:47.617383 14335 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.617439 14335 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:47.617463 14335 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.625702 14335 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.70:33933
I20240501 19:56:47.625747 14462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.70:33933 every 8 connection(s)
I20240501 19:56:47.626611 14335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 19:56:47.630461 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 14335
I20240501 19:56:47.630861 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.71:46083
--local_ip_for_outbound_sockets=127.5.139.71
--tserver_master_addrs=127.5.139.126:35403
--webserver_port=38429
--webserver_interface=127.5.139.71
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:47.637996 14463 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:47.638228 14463 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:47.638881 14463 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:47.640020 12449 ts_manager.cc:194] Re-registered known tserver with Master: 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933)
I20240501 19:56:47.640808 12449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.70:34383
W20240501 19:56:47.742025 14466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:47.742246 14466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:47.742290 14466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:47.745524 14466 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:47.745606 14466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:47.745700 14466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.71
I20240501 19:56:47.749583 14466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.71:46083
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.5.139.71
--webserver_port=38429
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.14466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.71
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:47.750540 14466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:47.751622 14466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:47.759900 14472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.760423 14475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.761108 14473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:47.761809 14466 server_base.cc:1034] running on GCE node
I20240501 19:56:47.762238 14466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:47.762849 14466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:47.764025 14466 hybrid_clock.cc:648] HybridClock initialized: now 1714593407763997 us; error 43 us; skew 500 ppm
I20240501 19:56:47.766050 14466 webserver.cc:460] Webserver started at http://127.5.139.71:38429/ using document root <none> and password file <none>
I20240501 19:56:47.766589 14466 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:47.766659 14466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:47.769845 14466 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:47.771790 14481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.772891 14466 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 19:56:47.772989 14466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "1d49e80be3d04fb99b711c62d61bac09"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:47.773474 14466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:47.798650 14466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:47.799192 14466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:47.799321 14466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:47.799790 14466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:47.800594 14466 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:47.800639 14466 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.800678 14466 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:47.800705 14466 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.809687 14466 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.71:46083
I20240501 19:56:47.809782 14594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.71:46083 every 8 connection(s)
I20240501 19:56:47.810637 14466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 19:56:47.817273 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 14466
I20240501 19:56:47.817754 5677 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskuLQw07/build/debug/bin/kudu
/tmp/dist-test-taskuLQw07/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.139.72:37453
--local_ip_for_outbound_sockets=127.5.139.72
--tserver_master_addrs=127.5.139.126:35403
--webserver_port=38791
--webserver_interface=127.5.139.72
--builtin_ntp_servers=127.5.139.84:34051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 19:56:47.818850 14595 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:47.819062 14595 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:47.819716 14595 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:47.820673 12449 ts_manager.cc:194] Re-registered known tserver with Master: 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 19:56:47.821302 12449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.71:50829
W20240501 19:56:47.923022 14598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:47.923243 14598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 19:56:47.923311 14598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:47.926321 14598 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 19:56:47.926385 14598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 19:56:47.926476 14598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.139.72
I20240501 19:56:47.929980 14598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.139.84:34051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.139.72:37453
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.5.139.72
--webserver_port=38791
--tserver_master_addrs=127.5.139.126:35403
--never_fsync=true
--heap_profile_path=/tmp/kudu.14598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.139.72
--log_dir=/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type DEBUG
built by None at 01 May 2024 19:43:13 UTC on 1a15035a6830
build id 77
I20240501 19:56:47.930711 14598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 19:56:47.931699 14598 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 19:56:47.939750 14604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.940189 14606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 19:56:47.940790 14603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 19:56:47.941448 14598 server_base.cc:1034] running on GCE node
I20240501 19:56:47.941776 14598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 19:56:47.942302 14598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 19:56:47.943447 14598 hybrid_clock.cc:648] HybridClock initialized: now 1714593407943426 us; error 33 us; skew 500 ppm
I20240501 19:56:47.945160 14598 webserver.cc:460] Webserver started at http://127.5.139.72:38791/ using document root <none> and password file <none>
I20240501 19:56:47.945664 14598 fs_manager.cc:359] Metadata directory not provided
I20240501 19:56:47.945732 14598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 19:56:47.948740 14598 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20240501 19:56:47.950470 14612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.951382 14598 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:47.951469 14598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "c6d2c7096e404ea9a6f73b123d74c622"
format_stamp: "Formatted at 2024-05-01 19:56:38 on dist-test-slave-twc7"
I20240501 19:56:47.951849 14598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 19:56:47.969630 14598 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 19:56:47.970130 14598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 19:56:47.970259 14598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 19:56:47.970672 14598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 19:56:47.971504 14598 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 19:56:47.971552 14598 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.971612 14598 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 19:56:47.971639 14598 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 19:56:47.980064 14598 rpc_server.cc:307] RPC server started. Bound to: 127.5.139.72:37453
I20240501 19:56:47.980089 14725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.139.72:37453 every 8 connection(s)
I20240501 19:56:47.980988 14598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 19:56:47.983448 5677 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskuLQw07/build/debug/bin/kudu as pid 14598
I20240501 19:56:47.991633 14726 heartbeater.cc:347] Connected to a master server at 127.5.139.126:35403
I20240501 19:56:47.991865 14726 heartbeater.cc:464] Registering TS with master...
I20240501 19:56:47.992373 14726 heartbeater.cc:511] Master 127.5.139.126:35403 requested a full tablet report, sending...
I20240501 19:56:47.993402 12449 ts_manager.cc:194] Re-registered known tserver with Master: c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
I20240501 19:56:47.994184 12449 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.139.72:56963
I20240501 19:56:48.368831 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.388262 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.390172 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:48.406844 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.412686 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:48.427305 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.429898 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.433400 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:48.440874 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.447361 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.461107 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.461462 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.461623 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.471262 14332 heartbeater.cc:503] Master 127.5.139.126:35403 was elected leader, sending a full tablet report...
I20240501 19:56:48.481353 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.493484 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:48.517477 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.517545 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.525938 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.534868 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.540279 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.541471 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.545454 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:48.558781 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.560106 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:48.560384 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.565995 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.571435 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.579927 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.590118 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:48.592733 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.600775 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.603698 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:48.610507 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.616760 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.623818 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.626030 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.629253 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:48.647186 14463 heartbeater.cc:503] Master 127.5.139.126:35403 was elected leader, sending a full tablet report...
I20240501 19:56:48.666966 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.669126 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.674362 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:48.824342 14595 heartbeater.cc:503] Master 127.5.139.126:35403 was elected leader, sending a full tablet report...
I20240501 19:56:48.839569 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.841415 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.842726 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.853739 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.862200 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.862679 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:48.892235 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:48.898883 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.926738 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.939195 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.942627 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:48.948940 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.956579 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:48.963829 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.963868 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:48.982936 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:48.996141 14726 heartbeater.cc:503] Master 127.5.139.126:35403 was elected leader, sending a full tablet report...
I20240501 19:56:49.035254 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.037258 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.037726 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.042814 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.043315 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.044407 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.052587 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.065570 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.069833 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.072743 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.078310 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.081743 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.082031 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.084182 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.085762 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.086063 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.086150 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.118463 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.127377 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.130251 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.138674 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.165483 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.180084 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.189474 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.299435 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.307258 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.309466 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.326782 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.329319 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.340111 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.353019 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.353744 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.399538 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.408466 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.413547 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.416313 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.425765 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.427979 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.441838 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.450168 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.482533 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.509754 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.519202 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.523420 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.523850 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.529732 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.557264 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.561889 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.563555 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.568941 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.570145 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.577661 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.589429 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.596182 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.596287 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.623477 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.650085 12726 consensus_queue.cc:237] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:49.652637 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.652909 12856 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.653540 12988 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.654237 13712 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.655782 15039 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.660813 12726 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:49.663131 12854 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.663230 12987 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.663640 13712 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.663976 13712 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.668872 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.672120 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.681211 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.681602 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.691040 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:49.697875 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.699610 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.700507 15039 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.703025 12856 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.705610 12445 catalog_manager.cc:5484] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:49.706951 12986 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.709537 12986 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:49.712620 12856 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.713124 14177 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.713657 12595 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.714337 14177 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:49.720487 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 851eeb0ab3364529907d9d09f2a4022e. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:49.720526 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 800d5f43cd1b4fb6bd8bbcac262aeb80. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:49.725095 12661 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e84a3865ce64eaebad4df46ec01842d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.726015 14177 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.728361 12595 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.731448 12452 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.736637 12856 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.743856 15039 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.744318 12854 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.744751 12726 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:49.746680 12986 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.747517 15472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.747920 12856 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.748299 13712 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.748689 12986 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.751914 13712 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:49.752236 12987 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
W20240501 19:56:49.758057 12661 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4b15761177f4cffb7aa5e4d16e42ac7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.758579 12987 consensus_queue.cc:237] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:49.760013 12595 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.760385 15488 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.761030 12726 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.761910 15500 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:49.762370 12924 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08535cb2f46464b9e510a45fee67b4f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.763192 12856 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:49.764065 12444 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.765508 15488 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.767545 12726 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.770619 12595 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.772785 12445 catalog_manager.cc:5484] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.774412 12452 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:49.779641 12726 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:49.781738 12595 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.782120 15039 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.782940 12987 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.784806 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.784875 12856 consensus_queue.cc:237] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:49.787324 12988 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.787689 14144 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:49.789528 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b61e94fc7e541c8aa84957560508a31. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.790158 12595 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.790540 14144 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.793390 14144 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.795925 12444 catalog_manager.cc:5484] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:49.798296 12595 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.798596 15472 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.801462 12988 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.801761 12595 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.802088 12445 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:49.812778 12595 consensus_queue.cc:237] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:49.814554 12726 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.814563 12856 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.815018 14467 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.815366 14467 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:49.815411 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fb481d394034a8f85af613beb5c9ec8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.816282 12988 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:49.817845 12595 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.818321 14177 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.820605 12854 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.821920 14177 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.822477 12988 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:49.824004 14467 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
W20240501 19:56:49.824970 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 706878734216445f8422e422228d4230. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.826125 12854 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.826545 12726 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.826995 12445 catalog_manager.cc:5484] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:49.833784 12988 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
W20240501 19:56:49.834124 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d940b16d07d54945abe4a10a11dd6a2a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.835748 12594 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:49.835846 12921 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 745e39ec23db4f90bb8b1e8b49a50244. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.836597 12726 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.837002 15508 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.837468 15508 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.839277 15508 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.839627 12854 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.840953 12595 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.841689 12452 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:49.844539 15500 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.844820 12594 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.849689 12444 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.856428 12726 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.864522 12988 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:49.866001 12594 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:49.866503 12854 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:49.866660 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 706878734216445f8422e422228d4230. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.866844 14177 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:49.868278 15500 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:49.869612 15500 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:49.870396 12854 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
W20240501 19:56:49.871788 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 706878734216445f8422e422228d4230. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.871645 12594 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:49.872344 12444 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:49.888960 12594 consensus_queue.cc:237] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:49.890913 12726 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.891567 15568 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:49.896049 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22b272a8582d4060a6a527ac4fce9ae5. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.901566 12988 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.901943 15568 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.917201 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.922169 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.923892 15568 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.927821 12988 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.930027 12445 catalog_manager.cc:5484] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:49.934933 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.936848 12726 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:49.944669 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:49.957243 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:49.963686 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.966197 12854 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:49.968318 12726 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.969177 14144 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:49.970587 12988 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:49.971554 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:49.972287 14144 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:49.972927 12790 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04ccc8addc55441b9ab9d8265b90b03b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:49.977238 14144 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.979414 12726 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.979553 12988 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.982239 12450 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:49.999940 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:50.024960 12854 consensus_queue.cc:237] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.026742 12594 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.027242 12988 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.027340 15542 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.029331 15544 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:50.031632 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04a70a2ee2144f2287795ffe723d2716. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.032370 15542 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:50.035037 12594 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:50.035166 12452 catalog_manager.cc:5484] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.040689 12988 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:50.066237 12594 consensus_queue.cc:237] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.068195 12854 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.068660 12726 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:50.069022 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db240ec5f8b2444fb6114ad814f3cdb7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.069361 15654 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.069720 15569 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.073401 15726 ts_tablet_manager.cc:902] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:50.074254 15726 tablet_copy_client.cc:308] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:50.075454 12725 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.077062 14467 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:50.077253 12988 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.078320 12854 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:50.078349 12659 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e84a3865ce64eaebad4df46ec01842d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.079161 12726 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:50.080657 13712 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:56:50.081481 12661 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e84a3865ce64eaebad4df46ec01842d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.081914 13712 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.082828 12450 catalog_manager.cc:5484] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.083925 13712 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.085350 12854 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:50.086303 12988 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.090505 12450 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.091519 12854 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.100095 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:47601)
I20240501 19:56:50.100620 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:47601: session id = 9069ec6fcdb248479037f2d37e0e851d-800d5f43cd1b4fb6bd8bbcac262aeb80
I20240501 19:56:50.102221 12746 tablet_copy_source_session.cc:205] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.102684 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.106333 15726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.111362 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:50.116863 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.121712 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.121838 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:50.126073 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:50.132426 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.133970 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.136672 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.147856 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:50.155001 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.157375 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.184322 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.193058 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.205593 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.217092 15726 tablet_copy_client.cc:783] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.217450 15726 tablet_copy_client.cc:647] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.218432 15788 ts_tablet_manager.cc:902] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:50.218737 15789 ts_tablet_manager.cc:902] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:50.219517 15789 tablet_copy_client.cc:308] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:50.224048 15788 tablet_copy_client.cc:308] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:50.224678 15726 tablet_copy_client.cc:523] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.227108 15726 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:50.228595 15800 ts_tablet_manager.cc:902] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:50.229245 15800 tablet_copy_client.cc:308] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:50.229745 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.235023 13008 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 851eeb0ab3364529907d9d09f2a4022e from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:43047)
I20240501 19:56:50.235162 13008 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 851eeb0ab3364529907d9d09f2a4022e from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:43047: session id = 1d49e80be3d04fb99b711c62d61bac09-851eeb0ab3364529907d9d09f2a4022e
I20240501 19:56:50.236562 13008 tablet_copy_source_session.cc:205] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.237514 12988 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.239512 12854 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.240399 15788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851eeb0ab3364529907d9d09f2a4022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.240782 14177 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:56:50.241236 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 851eeb0ab3364529907d9d09f2a4022e. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:50.242936 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 851eeb0ab3364529907d9d09f2a4022e. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.242897 14177 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.246379 12854 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.250416 12594 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.250336 12449 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.250909 15508 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.252758 12594 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.253460 15788 tablet_copy_client.cc:783] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.253446 15726 log.cc:824] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:50.254423 15788 tablet_copy_client.cc:647] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.257346 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 8b61e94fc7e541c8aa84957560508a31 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:41689)
I20240501 19:56:50.257450 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 8b61e94fc7e541c8aa84957560508a31 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:41689: session id = 1d49e80be3d04fb99b711c62d61bac09-8b61e94fc7e541c8aa84957560508a31
I20240501 19:56:50.257676 15806 ts_tablet_manager.cc:902] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:50.257863 15807 ts_tablet_manager.cc:902] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:50.258363 12746 tablet_copy_source_session.cc:205] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.259197 15800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b61e94fc7e541c8aa84957560508a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.261139 12594 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.262035 15800 tablet_copy_client.cc:783] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.262284 15800 tablet_copy_client.cc:647] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.262993 15788 tablet_copy_client.cc:523] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.263270 12725 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.263729 15654 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.265470 15788 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:50.266024 12988 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.267057 15654 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:50.268538 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d7ea564ea504163a1e17561be9a5130. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.272190 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.279317 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.286176 15827 ts_tablet_manager.cc:902] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:50.286852 15827 tablet_copy_client.cc:308] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:50.289966 13008 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet f08535cb2f46464b9e510a45fee67b4f from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:43047)
I20240501 19:56:50.290072 13008 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet f08535cb2f46464b9e510a45fee67b4f from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:43047: session id = 1d49e80be3d04fb99b711c62d61bac09-f08535cb2f46464b9e510a45fee67b4f
I20240501 19:56:50.291081 13008 tablet_copy_source_session.cc:205] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.291811 15800 tablet_copy_client.cc:523] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.294143 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08535cb2f46464b9e510a45fee67b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.296811 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:50.296173 15827 tablet_copy_client.cc:783] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.297094 15827 tablet_copy_client.cc:647] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.298668 15800 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:50.303007 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.308323 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.311233 15726 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.311653 15726 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:50.312011 15726 ts_tablet_manager.cc:1367] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.085s user 0.038s sys 0.003s
I20240501 19:56:50.315187 15800 log.cc:824] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:50.315043 15726 raft_consensus.cc:348] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.315289 15726 raft_consensus.cc:729] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:50.315666 15726 consensus_queue.cc:260] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.316586 15726 ts_tablet_manager.cc:1398] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240501 19:56:50.321012 15835 ts_tablet_manager.cc:902] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:50.324359 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-800d5f43cd1b4fb6bd8bbcac262aeb80 received from {username='slave'} at 127.5.139.69:47601
I20240501 19:56:50.324472 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-800d5f43cd1b4fb6bd8bbcac262aeb80 on tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:50.325850 15806 tablet_copy_client.cc:308] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:50.327389 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 0fb481d394034a8f85af613beb5c9ec8 from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:37741)
I20240501 19:56:50.327548 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 0fb481d394034a8f85af613beb5c9ec8 from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:37741: session id = c6d2c7096e404ea9a6f73b123d74c622-0fb481d394034a8f85af613beb5c9ec8
I20240501 19:56:50.327688 15807 tablet_copy_client.cc:308] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:50.329212 12877 tablet_copy_source_session.cc:205] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.334794 15835 tablet_copy_client.cc:308] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:50.335283 15789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb481d394034a8f85af613beb5c9ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.337142 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.338088 13008 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 745e39ec23db4f90bb8b1e8b49a50244 from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:43093)
I20240501 19:56:50.338209 13008 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 745e39ec23db4f90bb8b1e8b49a50244 from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:43093: session id = 9069ec6fcdb248479037f2d37e0e851d-745e39ec23db4f90bb8b1e8b49a50244
I20240501 19:56:50.338598 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 706878734216445f8422e422228d4230 from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:43093)
I20240501 19:56:50.338685 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 706878734216445f8422e422228d4230 from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:43093: session id = 9069ec6fcdb248479037f2d37e0e851d-706878734216445f8422e422228d4230
I20240501 19:56:50.339107 13008 tablet_copy_source_session.cc:205] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.340128 13007 tablet_copy_source_session.cc:205] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.341084 15835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706878734216445f8422e422228d4230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.343818 15827 tablet_copy_client.cc:523] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.344224 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.345603 15827 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:50.346377 15789 tablet_copy_client.cc:783] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.346635 15789 tablet_copy_client.cc:647] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.348881 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:50.349737 15835 tablet_copy_client.cc:783] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.349926 15835 tablet_copy_client.cc:647] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.357533 15807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745e39ec23db4f90bb8b1e8b49a50244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.360661 15807 tablet_copy_client.cc:783] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.360857 15807 tablet_copy_client.cc:647] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.369844 15856 ts_tablet_manager.cc:902] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:50.372077 15569 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.373028 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet e4b15761177f4cffb7aa5e4d16e42ac7 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:36001)
I20240501 19:56:50.373189 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet e4b15761177f4cffb7aa5e4d16e42ac7 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:36001: session id = 2ca06dad25a645bca71b09741798194a-e4b15761177f4cffb7aa5e4d16e42ac7
I20240501 19:56:50.374094 12746 tablet_copy_source_session.cc:205] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.377537 12725 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.378139 15856 tablet_copy_client.cc:308] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:50.375059 12444 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.384085 15789 tablet_copy_client.cc:523] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.384738 12725 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.386430 15789 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
W20240501 19:56:50.386809 12661 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bc9fe9ade8246779164d6db31b11e16. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.387039 12595 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.387410 15039 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.388072 15851 ts_tablet_manager.cc:902] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:50.388525 15851 tablet_copy_client.cc:308] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:50.389153 12988 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.391422 15835 tablet_copy_client.cc:523] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.391466 15807 tablet_copy_client.cc:523] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.392805 15835 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:50.393932 15807 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:50.396186 12988 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.397333 13712 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.401014 12615 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet d940b16d07d54945abe4a10a11dd6a2a from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:42155)
I20240501 19:56:50.401186 12615 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet d940b16d07d54945abe4a10a11dd6a2a from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:42155: session id = c6d2c7096e404ea9a6f73b123d74c622-d940b16d07d54945abe4a10a11dd6a2a
I20240501 19:56:50.402516 12615 tablet_copy_source_session.cc:205] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.404069 15851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940b16d07d54945abe4a10a11dd6a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.406510 15851 tablet_copy_client.cc:783] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.406826 15851 tablet_copy_client.cc:647] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.409441 15851 tablet_copy_client.cc:523] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.410705 15851 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:50.411365 15800 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.412968 15806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b15761177f4cffb7aa5e4d16e42ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.413216 15875 ts_tablet_manager.cc:902] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:50.413786 15875 tablet_copy_client.cc:308] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:50.414273 12615 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet 22b272a8582d4060a6a527ac4fce9ae5 from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:42155)
I20240501 19:56:50.414371 12615 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet 22b272a8582d4060a6a527ac4fce9ae5 from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:42155: session id = c6d2c7096e404ea9a6f73b123d74c622-22b272a8582d4060a6a527ac4fce9ae5
I20240501 19:56:50.414578 15800 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:50.414856 15800 ts_tablet_manager.cc:1367] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.116s user 0.033s sys 0.002s
I20240501 19:56:50.415259 12615 tablet_copy_source_session.cc:205] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.416011 15875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22b272a8582d4060a6a527ac4fce9ae5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.417459 15800 raft_consensus.cc:348] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.418859 15875 tablet_copy_client.cc:783] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.418970 15800 raft_consensus.cc:729] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:50.419260 15875 tablet_copy_client.cc:647] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.419348 15800 consensus_queue.cc:260] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.420423 15851 log.cc:824] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:50.420836 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 706878734216445f8422e422228d4230 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:50147)
I20240501 19:56:50.420946 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 706878734216445f8422e422228d4230 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:50147: session id = 2ca06dad25a645bca71b09741798194a-706878734216445f8422e422228d4230
I20240501 19:56:50.421775 15800 ts_tablet_manager.cc:1398] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240501 19:56:50.421907 13007 tablet_copy_source_session.cc:205] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.422704 15856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706878734216445f8422e422228d4230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.426513 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-8b61e94fc7e541c8aa84957560508a31 received from {username='slave'} at 127.5.139.71:41689
I20240501 19:56:50.426592 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-8b61e94fc7e541c8aa84957560508a31 on tablet 8b61e94fc7e541c8aa84957560508a31 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:50.431627 15856 tablet_copy_client.cc:783] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.431887 15856 tablet_copy_client.cc:647] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.433382 15856 tablet_copy_client.cc:523] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.435020 13712 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.435346 15856 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:50.437707 15835 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.438011 15835 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:50.438225 15835 ts_tablet_manager.cc:1367] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.045s user 0.026s sys 0.000s
I20240501 19:56:50.438544 12595 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.438877 12453 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.442209 15835 raft_consensus.cc:348] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.442438 15835 raft_consensus.cc:729] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:50.442602 15835 consensus_queue.cc:260] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.444224 15875 tablet_copy_client.cc:523] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.445581 15875 tablet_bootstrap.cc:492] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:50.446162 15835 ts_tablet_manager.cc:1398] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:50.447113 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-706878734216445f8422e422228d4230 received from {username='slave'} at 127.5.139.69:43093
I20240501 19:56:50.447232 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-706878734216445f8422e422228d4230 on tablet 706878734216445f8422e422228d4230 with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:50.451442 12988 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.454299 15806 tablet_copy_client.cc:783] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.454493 15806 tablet_copy_client.cc:647] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.458144 15806 tablet_copy_client.cc:523] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.458362 12595 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.459172 15856 log.cc:824] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Log is configured to *not* fsync() on all Append() calls
I20240501 19:56:50.460532 15806 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:50.460717 12854 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.463413 15568 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:50.467257 12528 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf6948589837482a9f580731e3b861b8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.467571 12988 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.468017 15568 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.471184 12725 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.472870 15568 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.473747 14549 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.474838 12987 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.474918 13712 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.475144 12595 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:50.475574 12661 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b61e94fc7e541c8aa84957560508a31. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.476074 12854 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.476213 13712 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.477869 15553 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.477994 12452 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.478328 15553 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.480937 12453 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.481209 12987 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.485302 12595 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.493361 12988 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.499099 15806 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.499536 15806 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:50.499902 15806 ts_tablet_manager.cc:1367] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.039s user 0.032s sys 0.000s
I20240501 19:56:50.503176 15039 raft_consensus.cc:1050] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:50.503049 15806 raft_consensus.cc:348] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.503343 15806 raft_consensus.cc:729] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:50.503635 15039 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.503692 15806 consensus_queue.cc:260] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.504590 15806 ts_tablet_manager.cc:1398] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 19:56:50.505939 12595 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:50.506377 12987 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:50.507651 14549 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
W20240501 19:56:50.509960 12661 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b61e94fc7e541c8aa84957560508a31. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.510285 12854 consensus_queue.cc:237] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.510417 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-e4b15761177f4cffb7aa5e4d16e42ac7 received from {username='slave'} at 127.5.139.70:36001
I20240501 19:56:50.510483 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2ca06dad25a645bca71b09741798194a-e4b15761177f4cffb7aa5e4d16e42ac7 on tablet e4b15761177f4cffb7aa5e4d16e42ac7 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:50.510731 15553 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:50.511386 13712 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:50.512734 12595 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.513942 15542 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.515482 15542 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.517453 12593 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.518949 12988 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:50.519105 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.519400 15678 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.520354 12453 catalog_manager.cc:5484] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:50.521379 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fb481d394034a8f85af613beb5c9ec8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.521636 15553 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.522161 12987 consensus_queue.cc:237] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.523042 12988 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.524339 12595 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
W20240501 19:56:50.525208 12924 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:50.525673 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08535cb2f46464b9e510a45fee67b4f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.526130 12725 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.526773 14548 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:50.526921 12453 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.528515 14177 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.529480 15907 ts_tablet_manager.cc:902] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:50.529855 15039 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:50.530068 15907 tablet_copy_client.cc:308] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:50.530817 14177 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.531167 12615 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet db240ec5f8b2444fb6114ad814f3cdb7 from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:42155)
I20240501 19:56:50.531265 12615 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet db240ec5f8b2444fb6114ad814f3cdb7 from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:42155: session id = c6d2c7096e404ea9a6f73b123d74c622-db240ec5f8b2444fb6114ad814f3cdb7
I20240501 19:56:50.532110 12615 tablet_copy_source_session.cc:205] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.533674 12444 catalog_manager.cc:5484] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.533803 14548 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.535809 12987 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.537725 15800 ts_tablet_manager.cc:902] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:50.538177 15800 tablet_copy_client.cc:308] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:50.538362 12725 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.540069 15907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db240ec5f8b2444fb6114ad814f3cdb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.540720 12595 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.541597 15500 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.543260 12595 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.544072 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 04a70a2ee2144f2287795ffe723d2716 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:50211)
I20240501 19:56:50.544176 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 04a70a2ee2144f2287795ffe723d2716 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:50211: session id = 1d49e80be3d04fb99b711c62d61bac09-04a70a2ee2144f2287795ffe723d2716
I20240501 19:56:50.545197 12877 tablet_copy_source_session.cc:205] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.546314 15800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a70a2ee2144f2287795ffe723d2716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.546366 15907 tablet_copy_client.cc:783] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.546519 15907 tablet_copy_client.cc:647] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.547740 15907 tablet_copy_client.cc:523] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.548455 15800 tablet_copy_client.cc:783] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.548622 15800 tablet_copy_client.cc:647] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.549553 15907 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:50.551962 15800 tablet_copy_client.cc:523] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.553061 15800 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:50.560756 12725 consensus_queue.cc:237] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.563236 12854 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.563356 15835 ts_tablet_manager.cc:902] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
W20240501 19:56:50.563557 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32a17c2e392a49b7b181fa7a25a778a1. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.563761 15835 tablet_copy_client.cc:308] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:50.563832 12593 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.564225 13712 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.565629 15472 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.565984 15788 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.566306 15788 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:50.566485 15788 ts_tablet_manager.cc:1367] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.301s user 0.030s sys 0.000s
I20240501 19:56:50.567077 15788 raft_consensus.cc:348] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.567202 15788 raft_consensus.cc:729] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:50.567304 15788 consensus_queue.cc:260] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.567555 15788 ts_tablet_manager.cc:1398] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:50.568450 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-851eeb0ab3364529907d9d09f2a4022e received from {username='slave'} at 127.5.139.71:43047
I20240501 19:56:50.568536 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-851eeb0ab3364529907d9d09f2a4022e on tablet 851eeb0ab3364529907d9d09f2a4022e with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:50.573992 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 04ccc8addc55441b9ab9d8265b90b03b from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:59875)
I20240501 19:56:50.574093 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 04ccc8addc55441b9ab9d8265b90b03b from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:59875: session id = 9069ec6fcdb248479037f2d37e0e851d-04ccc8addc55441b9ab9d8265b90b03b
I20240501 19:56:50.574309 12725 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.574990 12877 tablet_copy_source_session.cc:205] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.575685 15827 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.575739 12593 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.575950 12856 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.576136 13712 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.576361 15835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ccc8addc55441b9ab9d8265b90b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.577011 15827 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:50.578029 15827 ts_tablet_manager.cc:1367] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.232s user 0.028s sys 0.002s
I20240501 19:56:50.578500 15835 tablet_copy_client.cc:783] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.578626 15835 tablet_copy_client.cc:647] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
W20240501 19:56:50.579306 12659 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e4d79f5e11b4923acb2ce543884b54a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.579453 13712 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.580034 15726 ts_tablet_manager.cc:902] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:50.580461 15726 tablet_copy_client.cc:308] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:50.580883 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 3e84a3865ce64eaebad4df46ec01842d from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:47601)
I20240501 19:56:50.580966 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 3e84a3865ce64eaebad4df46ec01842d from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:47601: session id = 9069ec6fcdb248479037f2d37e0e851d-3e84a3865ce64eaebad4df46ec01842d
I20240501 19:56:50.581866 12746 tablet_copy_source_session.cc:205] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.582593 15726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e84a3865ce64eaebad4df46ec01842d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.584430 15726 tablet_copy_client.cc:783] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.584556 15726 tablet_copy_client.cc:647] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.585467 15827 raft_consensus.cc:348] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.588817 15827 raft_consensus.cc:729] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:50.589001 15827 consensus_queue.cc:260] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.589675 15835 tablet_copy_client.cc:523] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.589682 13712 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:50.591149 15835 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:50.591183 15827 ts_tablet_manager.cc:1398] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240501 19:56:50.592344 12452 catalog_manager.cc:5484] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.593529 12595 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:50.594667 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-f08535cb2f46464b9e510a45fee67b4f received from {username='slave'} at 127.5.139.71:43047
I20240501 19:56:50.594765 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-f08535cb2f46464b9e510a45fee67b4f on tablet f08535cb2f46464b9e510a45fee67b4f with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:50.597739 15925 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.599792 12854 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:50.600180 12595 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.602988 12453 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.609866 12856 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.613435 15851 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.613619 15726 tablet_copy_client.cc:523] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.614168 15851 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:50.613816 15875 tablet_bootstrap.cc:492] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.614605 15875 tablet_bootstrap.cc:492] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:50.614874 15875 ts_tablet_manager.cc:1367] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.169s user 0.024s sys 0.006s
I20240501 19:56:50.615082 15726 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:50.617624 15875 raft_consensus.cc:348] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.621306 15807 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.636826 15875 raft_consensus.cc:729] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:50.637426 15875 consensus_queue.cc:260] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.638002 15807 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:50.638257 15807 ts_tablet_manager.cc:1367] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.244s user 0.026s sys 0.004s
I20240501 19:56:50.621548 15851 ts_tablet_manager.cc:1367] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.211s user 0.022s sys 0.012s
I20240501 19:56:50.638975 15851 raft_consensus.cc:348] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.639163 15851 raft_consensus.cc:729] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:50.639302 15851 consensus_queue.cc:260] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.640311 15851 ts_tablet_manager.cc:1398] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:50.642066 15875 ts_tablet_manager.cc:1398] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.027s user 0.004s sys 0.000s
I20240501 19:56:50.642271 12615 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-d940b16d07d54945abe4a10a11dd6a2a received from {username='slave'} at 127.5.139.72:42155
I20240501 19:56:50.642367 12615 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-d940b16d07d54945abe4a10a11dd6a2a on tablet d940b16d07d54945abe4a10a11dd6a2a with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:50.642863 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-22b272a8582d4060a6a527ac4fce9ae5 received from {username='slave'} at 127.5.139.72:42155
I20240501 19:56:50.642935 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-22b272a8582d4060a6a527ac4fce9ae5 on tablet 22b272a8582d4060a6a527ac4fce9ae5 with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:50.643316 15807 raft_consensus.cc:348] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.648247 15807 raft_consensus.cc:729] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:50.648415 15807 consensus_queue.cc:260] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.649943 15807 ts_tablet_manager.cc:1398] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:50.650466 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-745e39ec23db4f90bb8b1e8b49a50244 received from {username='slave'} at 127.5.139.69:43093
I20240501 19:56:50.650569 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-745e39ec23db4f90bb8b1e8b49a50244 on tablet 745e39ec23db4f90bb8b1e8b49a50244 with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:50.652153 12595 consensus_queue.cc:237] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.653002 15789 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.653388 15789 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:50.654162 12856 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.654275 12725 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.654620 15789 ts_tablet_manager.cc:1367] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.268s user 0.033s sys 0.000s
I20240501 19:56:50.654593 15569 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:50.654758 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 544caa38514643d5b81cd6ed85e0bae7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.655289 15654 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.657374 15856 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.655351 15789 raft_consensus.cc:348] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.657951 15789 raft_consensus.cc:729] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:50.658162 15789 consensus_queue.cc:260] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.661942 15789 ts_tablet_manager.cc:1398] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:50.662552 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-0fb481d394034a8f85af613beb5c9ec8 received from {username='slave'} at 127.5.139.72:37741
I20240501 19:56:50.662663 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-0fb481d394034a8f85af613beb5c9ec8 on tablet 0fb481d394034a8f85af613beb5c9ec8 with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:50.662689 15856 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:50.663698 15856 ts_tablet_manager.cc:1367] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.229s user 0.031s sys 0.002s
I20240501 19:56:50.664330 15856 raft_consensus.cc:348] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.664459 15856 raft_consensus.cc:729] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:50.664558 15856 consensus_queue.cc:260] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.668795 15654 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:50.670943 12856 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:50.671535 12452 catalog_manager.cc:5484] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.671994 12726 consensus_queue.cc:237] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.674608 14286 raft_consensus.cc:1203] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 19:56:50.674916 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12a50be843fb49ee8a66c9a17d271d61. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.675606 12595 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.676343 15827 tablet_replica.cc:330] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: stopping tablet replica
I20240501 19:56:50.675973 12987 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.676651 15827 raft_consensus.cc:2229] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 19:56:50.676599 12725 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:50.676863 15827 raft_consensus.cc:2258] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 19:56:50.676941 15472 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.678413 15827 ts_tablet_manager.cc:1875] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:50.678839 14286 raft_consensus.cc:1203] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:50.679670 15472 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.680281 15856 ts_tablet_manager.cc:1398] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240501 19:56:50.680667 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-706878734216445f8422e422228d4230 received from {username='slave'} at 127.5.139.70:50147
I20240501 19:56:50.680733 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 2ca06dad25a645bca71b09741798194a-706878734216445f8422e422228d4230 on tablet 706878734216445f8422e422228d4230 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:50.682231 15907 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.682835 15907 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:50.683022 15907 ts_tablet_manager.cc:1367] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.134s user 0.021s sys 0.004s
I20240501 19:56:50.683444 15827 ts_tablet_manager.cc:1888] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 19:56:50.683588 15827 log.cc:1196] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal/wals/851eeb0ab3364529907d9d09f2a4022e
I20240501 19:56:50.683955 15827 ts_tablet_manager.cc:902] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:50.684470 15827 tablet_copy_client.cc:308] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:50.683521 15907 raft_consensus.cc:348] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.685060 15907 raft_consensus.cc:729] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:50.685083 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 851eeb0ab3364529907d9d09f2a4022e from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:43047)
I20240501 19:56:50.685189 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 851eeb0ab3364529907d9d09f2a4022e from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:43047: session id = 1d49e80be3d04fb99b711c62d61bac09-851eeb0ab3364529907d9d09f2a4022e
I20240501 19:56:50.685189 15907 consensus_queue.cc:260] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.685544 15907 ts_tablet_manager.cc:1398] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:50.686050 13007 tablet_copy_source_session.cc:205] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.686899 15827 ts_tablet_manager.cc:1875] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:56:50.687227 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-db240ec5f8b2444fb6114ad814f3cdb7 received from {username='slave'} at 127.5.139.72:42155
I20240501 19:56:50.687302 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-db240ec5f8b2444fb6114ad814f3cdb7 on tablet db240ec5f8b2444fb6114ad814f3cdb7 with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:50.689035 15856 ts_tablet_manager.cc:902] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:50.689165 15827 ts_tablet_manager.cc:1888] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 19:56:50.689385 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851eeb0ab3364529907d9d09f2a4022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.689848 15856 tablet_copy_client.cc:308] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:50.690232 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 851eeb0ab3364529907d9d09f2a4022e from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:50147)
I20240501 19:56:50.690320 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 851eeb0ab3364529907d9d09f2a4022e from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:50147: session id = 2ca06dad25a645bca71b09741798194a-851eeb0ab3364529907d9d09f2a4022e
I20240501 19:56:50.691159 15827 tablet_copy_client.cc:783] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.691156 13007 tablet_copy_source_session.cc:205] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.691365 15827 tablet_copy_client.cc:647] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.691962 15856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851eeb0ab3364529907d9d09f2a4022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.695350 15800 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.695627 15800 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:50.695812 15800 ts_tablet_manager.cc:1367] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.143s user 0.024s sys 0.004s
I20240501 19:56:50.696256 15800 raft_consensus.cc:348] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.696388 15800 raft_consensus.cc:729] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:50.696504 15800 consensus_queue.cc:260] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:50.696671 15856 tablet_copy_client.cc:783] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.696832 15856 tablet_copy_client.cc:647] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.698525 15800 ts_tablet_manager.cc:1398] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:50.699095 15789 ts_tablet_manager.cc:902] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:50.699537 15789 tablet_copy_client.cc:308] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:50.702133 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-04a70a2ee2144f2287795ffe723d2716 received from {username='slave'} at 127.5.139.71:50211
I20240501 19:56:50.702226 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-04a70a2ee2144f2287795ffe723d2716 on tablet 04a70a2ee2144f2287795ffe723d2716 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:50.704056 15926 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.706403 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 3e84a3865ce64eaebad4df46ec01842d from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:60977)
I20240501 19:56:50.706506 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 3e84a3865ce64eaebad4df46ec01842d from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:60977: session id = c6d2c7096e404ea9a6f73b123d74c622-3e84a3865ce64eaebad4df46ec01842d
I20240501 19:56:50.707430 12746 tablet_copy_source_session.cc:205] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.707477 12987 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.708276 15789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e84a3865ce64eaebad4df46ec01842d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.710876 15789 tablet_copy_client.cc:783] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.711069 15789 tablet_copy_client.cc:647] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.711728 12595 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.712416 12452 catalog_manager.cc:5484] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.712770 15789 tablet_copy_client.cc:523] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.714022 15789 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:50.719223 15835 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.719503 15835 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:50.719679 15835 ts_tablet_manager.cc:1367] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.129s user 0.024s sys 0.000s
I20240501 19:56:50.720106 15835 raft_consensus.cc:348] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.720223 15835 raft_consensus.cc:729] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:50.720316 15835 consensus_queue.cc:260] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.723206 14417 raft_consensus.cc:1203] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:50.725494 14286 raft_consensus.cc:1203] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:50.727826 15827 tablet_copy_client.cc:523] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.729338 15827 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:50.730796 15856 tablet_copy_client.cc:523] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.732167 15856 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:50.733798 15726 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.734067 15726 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:50.734227 15726 ts_tablet_manager.cc:1367] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.119s user 0.024s sys 0.003s
I20240501 19:56:50.734490 15835 ts_tablet_manager.cc:1398] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.015s user 0.004s sys 0.000s
I20240501 19:56:50.735275 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-04ccc8addc55441b9ab9d8265b90b03b received from {username='slave'} at 127.5.139.69:59875
I20240501 19:56:50.735352 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-04ccc8addc55441b9ab9d8265b90b03b on tablet 04ccc8addc55441b9ab9d8265b90b03b with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:50.743264 15806 ts_tablet_manager.cc:902] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:50.743233 15726 raft_consensus.cc:348] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.743374 15726 raft_consensus.cc:729] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:50.743490 15726 consensus_queue.cc:260] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.743785 15806 tablet_copy_client.cc:308] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:50.746130 12856 consensus_queue.cc:237] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.747699 12595 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.748383 14548 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.748569 15789 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.748745 12987 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:50.748869 15789 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:50.749079 15789 ts_tablet_manager.cc:1367] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.003s
I20240501 19:56:50.749495 15542 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.749627 15789 raft_consensus.cc:348] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.749756 15789 raft_consensus.cc:729] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:50.750141 15726 ts_tablet_manager.cc:1398] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240501 19:56:50.750169 15544 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.750401 15544 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:50.750810 15789 consensus_queue.cc:260] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.751237 15789 ts_tablet_manager.cc:1398] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:50.751729 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-3e84a3865ce64eaebad4df46ec01842d received from {username='slave'} at 127.5.139.72:60977
I20240501 19:56:50.751818 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-3e84a3865ce64eaebad4df46ec01842d on tablet 3e84a3865ce64eaebad4df46ec01842d with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:50.752619 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-3e84a3865ce64eaebad4df46ec01842d received from {username='slave'} at 127.5.139.69:47601
I20240501 19:56:50.752701 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-3e84a3865ce64eaebad4df46ec01842d on tablet 3e84a3865ce64eaebad4df46ec01842d with peer 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.753237 12790 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04a70a2ee2144f2287795ffe723d2716. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.754467 15542 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.756971 12595 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.758723 15544 raft_consensus.cc:1050] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:50.759075 12987 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.759210 15544 consensus_queue.cc:237] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:50.760298 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.761896 12449 catalog_manager.cc:5484] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
W20240501 19:56:50.762753 12790 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04a70a2ee2144f2287795ffe723d2716. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.762992 12595 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:50.763372 14144 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:50.763489 14548 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.764020 12987 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:50.765619 14549 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:50.766654 15678 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:50.766927 15678 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:50.769526 15542 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.770910 15827 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.771188 15827 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:50.771355 15827 ts_tablet_manager.cc:1367] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.042s user 0.023s sys 0.004s
I20240501 19:56:50.771402 14549 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.771811 15827 raft_consensus.cc:348] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.771917 15827 raft_consensus.cc:729] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:50.772009 15827 consensus_queue.cc:260] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.772251 15827 ts_tablet_manager.cc:1398] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:50.773204 14417 raft_consensus.cc:1203] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:50.774418 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-851eeb0ab3364529907d9d09f2a4022e received from {username='slave'} at 127.5.139.71:43047
I20240501 19:56:50.774500 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-851eeb0ab3364529907d9d09f2a4022e on tablet 851eeb0ab3364529907d9d09f2a4022e with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:50.775465 12987 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.776423 12595 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.777261 12449 catalog_manager.cc:5484] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.782652 12856 consensus_queue.cc:237] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:50.784144 12987 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:50.784241 14549 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:50.784628 14144 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:50.784863 12595 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:50.784905 14144 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:50.785261 14144 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:50.787604 15542 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:50.787990 12593 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:50.789559 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04a70a2ee2144f2287795ffe723d2716 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:50.790064 14549 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:50.793815 12987 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:50.793891 12453 catalog_manager.cc:5484] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 6 to 7, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:50.799336 12856 consensus_queue.cc:237] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:50.800890 14549 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:50.801010 12987 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:50.801424 14144 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:50.801748 16064 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:50.803813 14144 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:50.805054 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04a70a2ee2144f2287795ffe723d2716 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:50.805747 14549 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:50.806054 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.806336 12453 catalog_manager.cc:5484] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.807634 12987 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:50.810914 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.813443 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a70a2ee2144f2287795ffe723d2716 with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:50.821875 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.833889 16072 tablet_replica.cc:330] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:50.836410 16072 raft_consensus.cc:2229] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:50.836834 16072 raft_consensus.cc:2258] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:50.836966 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:50.838395 16072 ts_tablet_manager.cc:1875] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:50.842013 16072 ts_tablet_manager.cc:1888] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:50.842157 16072 log.cc:1196] T 04a70a2ee2144f2287795ffe723d2716 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/04a70a2ee2144f2287795ffe723d2716
I20240501 19:56:50.843812 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 04a70a2ee2144f2287795ffe723d2716 (table rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]) successfully deleted
I20240501 19:56:50.845814 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.849598 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.861395 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet 7d7ea564ea504163a1e17561be9a5130 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:57697)
I20240501 19:56:50.861505 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet 7d7ea564ea504163a1e17561be9a5130 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:57697: session id = 2ca06dad25a645bca71b09741798194a-7d7ea564ea504163a1e17561be9a5130
I20240501 19:56:50.862466 12614 tablet_copy_source_session.cc:205] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.863431 15806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7ea564ea504163a1e17561be9a5130. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.865787 15806 tablet_copy_client.cc:783] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.865962 15806 tablet_copy_client.cc:647] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.866606 14266 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a70a2ee2144f2287795ffe723d2716 with delete_type TABLET_DATA_TOMBSTONED (TS 9069ec6fcdb248479037f2d37e0e851d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:54698
I20240501 19:56:50.867894 15806 tablet_copy_client.cc:523] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.869086 15806 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:50.890430 15856 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.890769 15856 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:50.890966 15856 ts_tablet_manager.cc:1367] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.159s user 0.023s sys 0.003s
I20240501 19:56:50.891440 15856 raft_consensus.cc:348] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.891585 15856 raft_consensus.cc:729] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:50.898030 12595 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:50.899508 12856 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.900142 15568 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:50.902069 12987 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:50.902541 15569 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:50.904472 12530 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 955edb7c12fa4b32ac58827f131931e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:50.905735 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:50.891713 15856 consensus_queue.cc:260] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.912441 15856 ts_tablet_manager.cc:1398] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20240501 19:56:50.912897 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-851eeb0ab3364529907d9d09f2a4022e received from {username='slave'} at 127.5.139.70:50147
I20240501 19:56:50.912986 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 2ca06dad25a645bca71b09741798194a-851eeb0ab3364529907d9d09f2a4022e on tablet 851eeb0ab3364529907d9d09f2a4022e with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:50.913261 15806 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:50.913811 15806 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:50.914000 15806 ts_tablet_manager.cc:1367] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.045s user 0.024s sys 0.005s
I20240501 19:56:50.914134 15856 ts_tablet_manager.cc:902] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:50.917819 14549 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:50.914435 15806 raft_consensus.cc:348] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.920538 15806 raft_consensus.cc:729] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:50.920686 15806 consensus_queue.cc:260] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.921113 15806 ts_tablet_manager.cc:1398] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 19:56:50.923403 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-7d7ea564ea504163a1e17561be9a5130 received from {username='slave'} at 127.5.139.70:57697
I20240501 19:56:50.923480 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session 2ca06dad25a645bca71b09741798194a-7d7ea564ea504163a1e17561be9a5130 on tablet 7d7ea564ea504163a1e17561be9a5130 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:50.926985 14680 raft_consensus.cc:1203] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 19:56:50.933187 12437 catalog_manager.cc:4797] TS 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): delete failed for tablet 04a70a2ee2144f2287795ffe723d2716 because the tablet was not found. No further retry: Not found: Tablet not found: 04a70a2ee2144f2287795ffe723d2716
I20240501 19:56:50.933341 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.934194 15569 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:50.934623 12856 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:50.935187 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.937675 15856 tablet_copy_client.cc:308] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:50.938392 12987 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:50.938758 12453 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:50.941676 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 8bc9fe9ade8246779164d6db31b11e16 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:36001)
I20240501 19:56:50.941778 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 8bc9fe9ade8246779164d6db31b11e16 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:36001: session id = 2ca06dad25a645bca71b09741798194a-8bc9fe9ade8246779164d6db31b11e16
I20240501 19:56:50.942833 12746 tablet_copy_source_session.cc:205] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.943562 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:50.944007 15856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9fe9ade8246779164d6db31b11e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.946197 15856 tablet_copy_client.cc:783] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.946363 15856 tablet_copy_client.cc:647] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.954648 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:50.962690 14679 raft_consensus.cc:1203] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:50.963533 14680 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:50.968466 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:50.980973 15726 ts_tablet_manager.cc:902] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:50.981456 15726 tablet_copy_client.cc:308] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:50.984262 15856 tablet_copy_client.cc:523] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:50.985795 15856 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:50.990413 14679 raft_consensus.cc:1203] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:50.993305 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet cf6948589837482a9f580731e3b861b8 from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:60521)
I20240501 19:56:50.993405 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet cf6948589837482a9f580731e3b861b8 from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:60521: session id = 9069ec6fcdb248479037f2d37e0e851d-cf6948589837482a9f580731e3b861b8
I20240501 19:56:50.994405 12614 tablet_copy_source_session.cc:205] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:50.995294 15726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6948589837482a9f580731e3b861b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:50.998544 15726 tablet_copy_client.cc:783] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:50.998733 15726 tablet_copy_client.cc:647] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:50.999905 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.005318 15726 tablet_copy_client.cc:523] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.006587 15726 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:51.009997 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:51.010633 12726 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:51.012814 12593 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:51.013331 12661 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.013633 15553 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.014127 15806 ts_tablet_manager.cc:902] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:51.014276 12987 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.014564 15806 tablet_copy_client.cc:308] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
W20240501 19:56:51.014763 12662 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bc9fe9ade8246779164d6db31b11e16. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.014861 15553 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.015308 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 8b61e94fc7e541c8aa84957560508a31 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:36001)
I20240501 19:56:51.015398 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 8b61e94fc7e541c8aa84957560508a31 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:36001: session id = 2ca06dad25a645bca71b09741798194a-8b61e94fc7e541c8aa84957560508a31
I20240501 19:56:51.016356 12746 tablet_copy_source_session.cc:205] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.016767 12593 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.019322 15806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b61e94fc7e541c8aa84957560508a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.016362 15553 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.024819 12987 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.025586 12451 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.028335 15806 tablet_copy_client.cc:783] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.028520 15806 tablet_copy_client.cc:647] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.033931 15827 ts_tablet_manager.cc:902] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:51.034435 15827 tablet_copy_client.cc:308] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:51.034869 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 32a17c2e392a49b7b181fa7a25a778a1 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:41689)
I20240501 19:56:51.034970 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 32a17c2e392a49b7b181fa7a25a778a1 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:41689: session id = 1d49e80be3d04fb99b711c62d61bac09-32a17c2e392a49b7b181fa7a25a778a1
I20240501 19:56:51.035941 12746 tablet_copy_source_session.cc:205] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.036818 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a17c2e392a49b7b181fa7a25a778a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.039177 15827 tablet_copy_client.cc:783] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.039362 15827 tablet_copy_client.cc:647] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.040073 14286 raft_consensus.cc:1203] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:51.041929 12726 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:51.043912 14549 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.044220 15835 ts_tablet_manager.cc:902] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:51.044570 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.044766 15835 tablet_copy_client.cc:308] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:51.045261 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 7e4d79f5e11b4923acb2ce543884b54a from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:47601)
I20240501 19:56:51.045356 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 7e4d79f5e11b4923acb2ce543884b54a from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:47601: session id = 9069ec6fcdb248479037f2d37e0e851d-7e4d79f5e11b4923acb2ce543884b54a
I20240501 19:56:51.045933 12593 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.046358 12746 tablet_copy_source_session.cc:205] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.046326 15925 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.046808 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.047303 15835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4d79f5e11b4923acb2ce543884b54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.048010 12988 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.048472 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.049556 15835 tablet_copy_client.cc:783] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.049726 15835 tablet_copy_client.cc:647] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.050546 15472 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.050740 15827 tablet_copy_client.cc:523] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.052255 15827 tablet_bootstrap.cc:492] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:51.052405 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8b61e94fc7e541c8aa84957560508a31 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.052953 12593 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.058632 15726 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.059104 15726 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:51.058972 12445 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.059306 15726 ts_tablet_manager.cc:1367] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.053s user 0.027s sys 0.001s
I20240501 19:56:51.059881 15726 raft_consensus.cc:348] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:51.060014 15726 raft_consensus.cc:729] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:51.060132 15726 consensus_queue.cc:260] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:51.062703 15726 ts_tablet_manager.cc:1398] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 19:56:51.069384 12987 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.070717 14549 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.071333 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-cf6948589837482a9f580731e3b861b8 received from {username='slave'} at 127.5.139.69:60521
I20240501 19:56:51.071413 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-cf6948589837482a9f580731e3b861b8 on tablet cf6948589837482a9f580731e3b861b8 with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:51.073141 16102 ts_tablet_manager.cc:902] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:51.073704 16102 tablet_copy_client.cc:308] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:51.074398 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 12a50be843fb49ee8a66c9a17d271d61 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:36001)
I20240501 19:56:51.074522 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 12a50be843fb49ee8a66c9a17d271d61 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:36001: session id = 2ca06dad25a645bca71b09741798194a-12a50be843fb49ee8a66c9a17d271d61
I20240501 19:56:51.075361 12746 tablet_copy_source_session.cc:205] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.076170 16102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a50be843fb49ee8a66c9a17d271d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.083719 12726 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:51.086364 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b61e94fc7e541c8aa84957560508a31 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca06dad25a645bca71b09741798194a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51964
I20240501 19:56:51.086952 14549 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.087510 13712 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.089263 15553 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.090005 14549 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.090899 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8b61e94fc7e541c8aa84957560508a31 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.083585 16102 tablet_copy_client.cc:783] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.094574 16102 tablet_copy_client.cc:647] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.094980 12593 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.095391 15039 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.096761 12453 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.098313 14286 raft_consensus.cc:1203] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 19:56:51.100543 12439 catalog_manager.cc:4810] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): delete failed for tablet 8b61e94fc7e541c8aa84957560508a31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b61e94fc7e541c8aa84957560508a31 already in progress: copying tablet
I20240501 19:56:51.102541 15806 tablet_copy_client.cc:523] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.104039 15806 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:51.104923 15726 ts_tablet_manager.cc:902] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:51.106220 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b61e94fc7e541c8aa84957560508a31 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:51.106967 15726 tablet_copy_client.cc:308] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:51.096889 12595 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.107429 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet f08535cb2f46464b9e510a45fee67b4f from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:43093)
I20240501 19:56:51.107522 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet f08535cb2f46464b9e510a45fee67b4f from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:43093: session id = 9069ec6fcdb248479037f2d37e0e851d-f08535cb2f46464b9e510a45fee67b4f
I20240501 19:56:51.108433 13007 tablet_copy_source_session.cc:205] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.109360 15726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f08535cb2f46464b9e510a45fee67b4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.111732 15726 tablet_copy_client.cc:783] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.111908 15726 tablet_copy_client.cc:647] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.115285 15835 tablet_copy_client.cc:523] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.116484 15835 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:51.117329 15827 tablet_bootstrap.cc:492] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.117688 15827 tablet_bootstrap.cc:492] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:51.117872 15827 ts_tablet_manager.cc:1367] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.066s user 0.029s sys 0.001s
I20240501 19:56:51.118314 15827 raft_consensus.cc:348] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:51.118438 15827 raft_consensus.cc:729] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:51.118541 15827 consensus_queue.cc:260] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:51.118809 15827 ts_tablet_manager.cc:1398] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:51.119235 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-32a17c2e392a49b7b181fa7a25a778a1 received from {username='slave'} at 127.5.139.71:41689
I20240501 19:56:51.119314 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-32a17c2e392a49b7b181fa7a25a778a1 on tablet 32a17c2e392a49b7b181fa7a25a778a1 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:51.123837 16179 tablet_replica.cc:330] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:51.124264 16179 raft_consensus.cc:2229] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.124550 16179 raft_consensus.cc:2258] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.126333 16179 ts_tablet_manager.cc:1875] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.135757 15726 tablet_copy_client.cc:523] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.137140 15726 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:51.140645 15508 raft_consensus.cc:1050] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:51.141184 15508 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.142834 14286 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.143316 14417 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.143920 15500 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.144376 12856 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.144994 15500 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.145273 15500 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.150054 12595 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.150645 15500 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.152801 13712 raft_consensus.cc:1050] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:51.152822 16102 tablet_copy_client.cc:523] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.153224 13712 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:51.154163 16102 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:51.154783 12854 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.154982 15835 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.155161 15553 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.155247 15835 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:51.155414 15835 ts_tablet_manager.cc:1367] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.039s user 0.028s sys 0.000s
I20240501 19:56:51.156360 15835 raft_consensus.cc:348] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:51.156476 15835 raft_consensus.cc:729] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:51.156555 15835 consensus_queue.cc:260] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:51.156581 12987 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.156932 15553 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.158362 14416 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.158818 15039 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.158872 15835 ts_tablet_manager.cc:1398] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 19:56:51.160921 16186 ts_tablet_manager.cc:902] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:51.161338 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-7e4d79f5e11b4923acb2ce543884b54a received from {username='slave'} at 127.5.139.69:47601
I20240501 19:56:51.161412 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-7e4d79f5e11b4923acb2ce543884b54a on tablet 7e4d79f5e11b4923acb2ce543884b54a with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:51.161652 16186 tablet_copy_client.cc:308] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:51.167183 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 0fb481d394034a8f85af613beb5c9ec8 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:54927)
I20240501 19:56:51.167275 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 0fb481d394034a8f85af613beb5c9ec8 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:54927: session id = 2ca06dad25a645bca71b09741798194a-0fb481d394034a8f85af613beb5c9ec8
I20240501 19:56:51.168169 12877 tablet_copy_source_session.cc:205] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.174153 12855 consensus_queue.cc:237] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:51.174948 16186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb481d394034a8f85af613beb5c9ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.175937 12987 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:51.176544 12726 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:51.176858 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bf64d9f3ce94db28b032cdcd795a255. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.177114 16064 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:51.178571 16064 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:51.178997 16179 ts_tablet_manager.cc:1888] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:51.179093 16179 log.cc:1196] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/8b61e94fc7e541c8aa84957560508a31
I20240501 19:56:51.179704 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 8b61e94fc7e541c8aa84957560508a31 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
I20240501 19:56:51.184592 16195 raft_consensus.cc:1013] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: attempt to promote peer 2ca06dad25a645bca71b09741798194a: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 19:56:51.185056 15856 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.185388 15856 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:51.185456 14417 raft_consensus.cc:1203] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:51.185551 15856 ts_tablet_manager.cc:1367] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.200s user 0.028s sys 0.001s
I20240501 19:56:51.186010 15856 raft_consensus.cc:348] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.186136 15856 raft_consensus.cc:729] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:51.186241 15856 consensus_queue.cc:260] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.186725 15508 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.189859 12856 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.190157 14286 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.191449 15806 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.191696 15806 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:51.191870 15806 ts_tablet_manager.cc:1367] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.088s user 0.025s sys 0.004s
I20240501 19:56:51.192580 15856 ts_tablet_manager.cc:1398] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 19:56:51.192838 15806 raft_consensus.cc:348] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.192939 15806 raft_consensus.cc:729] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:51.193140 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.193739 15806 consensus_queue.cc:260] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.194077 15806 ts_tablet_manager.cc:1398] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:51.194178 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-8bc9fe9ade8246779164d6db31b11e16 received from {username='slave'} at 127.5.139.70:36001
I20240501 19:56:51.194253 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2ca06dad25a645bca71b09741798194a-8bc9fe9ade8246779164d6db31b11e16 on tablet 8bc9fe9ade8246779164d6db31b11e16 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:51.194243 12445 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.194631 14417 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.194990 12745 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-8b61e94fc7e541c8aa84957560508a31 received from {username='slave'} at 127.5.139.70:36001
I20240501 19:56:51.195042 12745 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2ca06dad25a645bca71b09741798194a-8b61e94fc7e541c8aa84957560508a31 on tablet 8b61e94fc7e541c8aa84957560508a31 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:51.197513 16195 raft_consensus.cc:1050] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:51.197919 16195 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:51.199456 12856 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.199456 14286 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.199838 15488 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.200273 14417 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.200606 15508 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.201300 15500 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.203233 14416 raft_consensus.cc:1203] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:51.204186 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b61e94fc7e541c8aa84957560508a31 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 6 (current committed config index is 8)) from {username='slave'} at 127.0.0.1:51964
I20240501 19:56:51.204409 16178 tablet_replica.cc:330] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: stopping tablet replica
I20240501 19:56:51.204746 16178 raft_consensus.cc:2229] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Raft consensus shutting down.
I20240501 19:56:51.204669 15500 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.204962 16178 raft_consensus.cc:2258] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Raft consensus is shut down!
I20240501 19:56:51.206029 16178 ts_tablet_manager.cc:1875] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.207389 12856 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.208071 16178 ts_tablet_manager.cc:1888] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:51.208132 16178 log.cc:1196] T 8b61e94fc7e541c8aa84957560508a31 P 2ca06dad25a645bca71b09741798194a: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/wals/8b61e94fc7e541c8aa84957560508a31
I20240501 19:56:51.208611 12439 catalog_manager.cc:4830] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): tablet 8b61e94fc7e541c8aa84957560508a31 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
I20240501 19:56:51.209779 15488 raft_consensus.cc:1050] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:51.210041 12452 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.210165 15488 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.211594 12726 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.212114 14285 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.212510 16196 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.212854 16196 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.214080 16064 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:51.215533 15726 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.215806 15726 tablet_bootstrap.cc:492] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:51.215996 15726 ts_tablet_manager.cc:1367] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.079s user 0.027s sys 0.000s
I20240501 19:56:51.216240 12726 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:51.216444 15726 raft_consensus.cc:348] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:51.216554 15726 raft_consensus.cc:729] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:51.216656 15726 consensus_queue.cc:260] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:51.216915 15726 ts_tablet_manager.cc:1398] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:51.217921 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-f08535cb2f46464b9e510a45fee67b4f received from {username='slave'} at 127.5.139.69:43093
I20240501 19:56:51.217994 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-f08535cb2f46464b9e510a45fee67b4f on tablet f08535cb2f46464b9e510a45fee67b4f with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:51.219986 14549 raft_consensus.cc:1203] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:51.220541 12452 catalog_manager.cc:5484] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:51.221221 12594 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.222393 12987 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:51.224606 14417 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.226426 12593 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.228271 15553 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.228391 16186 tablet_copy_client.cc:783] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.228643 16186 tablet_copy_client.cc:647] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.230724 14416 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.231170 16186 tablet_copy_client.cc:523] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.231540 12854 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.232869 16186 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:51.233490 16196 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.233971 16195 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.234298 16195 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.235116 14285 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.235247 12595 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.236755 12987 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.237563 12593 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.238718 12726 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.238592 14286 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.238994 12595 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.241650 12451 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.246789 12453 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.258927 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:51.267900 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.269933 16102 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.270298 16102 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:51.270475 16102 ts_tablet_manager.cc:1367] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.116s user 0.028s sys 0.000s
I20240501 19:56:51.270587 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:51.270938 16102 raft_consensus.cc:348] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.271064 16102 raft_consensus.cc:729] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:51.271169 16102 consensus_queue.cc:260] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.271440 16102 ts_tablet_manager.cc:1398] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:51.272536 16186 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.272872 16186 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:51.273026 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.273111 16186 ts_tablet_manager.cc:1367] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.040s user 0.025s sys 0.002s
I20240501 19:56:51.273494 16186 raft_consensus.cc:348] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.273766 12745 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-12a50be843fb49ee8a66c9a17d271d61 received from {username='slave'} at 127.5.139.70:36001
I20240501 19:56:51.273850 12745 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2ca06dad25a645bca71b09741798194a-12a50be843fb49ee8a66c9a17d271d61 on tablet 12a50be843fb49ee8a66c9a17d271d61 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:51.274231 16186 raft_consensus.cc:729] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:51.277346 12987 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:51.278609 12988 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.279210 12595 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.279793 14286 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.279857 15500 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.280648 15488 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.281145 15039 raft_consensus.cc:1050] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:51.281133 14286 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.281538 15039 consensus_queue.cc:237] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.281805 15488 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.283361 12726 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.283597 12856 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.283691 14284 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.284044 15508 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.284142 15925 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.284288 15553 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.285408 15488 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.285950 12988 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.286271 15925 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.287335 15488 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.288995 14284 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.290729 12595 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.291445 12449 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 5 to 6, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.293428 12453 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.278609 14417 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.294343 12726 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.294759 15508 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.295857 16186 consensus_queue.cc:260] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.295925 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 706878734216445f8422e422228d4230 with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:51.296445 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 745e39ec23db4f90bb8b1e8b49a50244 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.296382 14417 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.296622 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 706878734216445f8422e422228d4230 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.296850 14284 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.297453 16186 ts_tablet_manager.cc:1398] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.024s user 0.000s sys 0.001s
I20240501 19:56:51.298126 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-0fb481d394034a8f85af613beb5c9ec8 received from {username='slave'} at 127.5.139.70:54927
I20240501 19:56:51.298207 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session 2ca06dad25a645bca71b09741798194a-0fb481d394034a8f85af613beb5c9ec8 on tablet 0fb481d394034a8f85af613beb5c9ec8 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:51.300288 14680 raft_consensus.cc:1203] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:51.303015 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.303292 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet 745e39ec23db4f90bb8b1e8b49a50244 with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:51.303527 16072 tablet_replica.cc:330] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:51.303679 16072 raft_consensus.cc:2229] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.303803 16072 raft_consensus.cc:2258] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.308157 16072 ts_tablet_manager.cc:1875] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.310395 16072 ts_tablet_manager.cc:1888] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:51.310467 16072 log.cc:1196] T 745e39ec23db4f90bb8b1e8b49a50244 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/745e39ec23db4f90bb8b1e8b49a50244
I20240501 19:56:51.310410 12726 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:51.311259 12988 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.311889 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 745e39ec23db4f90bb8b1e8b49a50244 (table rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]) successfully deleted
I20240501 19:56:51.311954 12595 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.312464 12854 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.312484 14284 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.313226 13712 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.313288 14284 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:51.313498 13712 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.313745 16195 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 19:56:51.314304 12662 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e4d79f5e11b4923acb2ce543884b54a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.314718 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.315850 12595 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:51.316169 16195 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:51.317579 16195 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.317634 13712 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.319757 14286 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.320675 12595 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.321098 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 706878734216445f8422e422228d4230 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.323129 12444 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.323431 12855 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.326290 15569 raft_consensus.cc:1050] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:51.326797 15569 consensus_queue.cc:237] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:51.329123 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 706878734216445f8422e422228d4230 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca06dad25a645bca71b09741798194a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51964
I20240501 19:56:51.329353 16178 tablet_replica.cc:330] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: stopping tablet replica
I20240501 19:56:51.329460 16178 raft_consensus.cc:2229] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.329586 16178 raft_consensus.cc:2258] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.330111 12726 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.330457 16178 ts_tablet_manager.cc:1875] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.330392 12444 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.330535 15654 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.330873 15553 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.332463 16178 ts_tablet_manager.cc:1888] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:51.332526 16178 log.cc:1196] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/wals/706878734216445f8422e422228d4230
I20240501 19:56:51.332973 12439 catalog_manager.cc:4830] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): tablet 706878734216445f8422e422228d4230 (table rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]) successfully deleted
I20240501 19:56:51.333456 14680 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.333731 12595 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.335438 12855 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.335827 15654 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.337703 15654 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.341886 15553 raft_consensus.cc:1050] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:51.342206 14284 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.342329 15553 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
W20240501 19:56:51.344996 12662 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e4d79f5e11b4923acb2ce543884b54a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.345742 12854 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.346163 13712 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.347767 14284 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.348261 15925 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.348439 12595 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.348920 13712 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.350791 12726 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:51.350839 15925 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.351588 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:51.353204 12854 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240501 19:56:51.353752 12662 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4b15761177f4cffb7aa5e4d16e42ac7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.354260 12988 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.354679 15472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.355548 15472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.356933 12854 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.357694 12595 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.360163 15472 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.361894 14417 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.361603 12453 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.362176 12857 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.362668 12988 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.366051 12449 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 5 to 6, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.366799 16267 tablet_replica.cc:330] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:51.367112 16267 raft_consensus.cc:2229] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.367343 16267 raft_consensus.cc:2258] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.371059 15654 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.371388 16267 ts_tablet_manager.cc:1875] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.372720 13712 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.373276 12725 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.375900 14680 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.376086 14417 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.379914 16267 ts_tablet_manager.cc:1888] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:51.380828 12444 catalog_manager.cc:5484] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.382004 16267 log.cc:1196] T 706878734216445f8422e422228d4230 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/706878734216445f8422e422228d4230
I20240501 19:56:51.382349 15553 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.382810 14285 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.384740 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 706878734216445f8422e422228d4230 (table rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]) successfully deleted
I20240501 19:56:51.390076 12988 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.393165 12444 catalog_manager.cc:5484] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.397745 14284 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.399701 12988 consensus_queue.cc:237] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.400780 16262 ts_tablet_manager.cc:902] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:51.400995 12725 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:51.401446 16262 tablet_copy_client.cc:308] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:51.402077 14549 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.402797 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet 544caa38514643d5b81cd6ed85e0bae7 from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:42155)
I20240501 19:56:51.402901 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet 544caa38514643d5b81cd6ed85e0bae7 from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:42155: session id = c6d2c7096e404ea9a6f73b123d74c622-544caa38514643d5b81cd6ed85e0bae7
I20240501 19:56:51.403029 12595 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.403447 15925 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.403436 12856 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.403373 14284 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.403870 12614 tablet_copy_source_session.cc:205] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.404132 15925 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.405153 16195 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:56:51.406592 12662 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e4d79f5e11b4923acb2ce543884b54a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.407099 12593 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.407269 12724 consensus_queue.cc:237] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
W20240501 19:56:51.407562 12923 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08535cb2f46464b9e510a45fee67b4f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.407582 12725 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.407780 16195 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.408284 16195 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.408505 15508 raft_consensus.cc:1013] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: attempt to promote peer 1d49e80be3d04fb99b711c62d61bac09: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 19:56:51.408990 14284 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.409201 14286 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.409497 16196 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.410002 15500 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.410506 12595 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.412684 16196 raft_consensus.cc:1050] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:51.412925 12451 catalog_manager.cc:5484] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.413079 16196 consensus_queue.cc:237] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.412926 14284 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.414942 12725 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.415344 14549 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.415527 15488 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.415598 14284 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:51.415947 12923 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08535cb2f46464b9e510a45fee67b4f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.416306 14548 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.416666 12725 raft_consensus.cc:858] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
W20240501 19:56:51.416729 12725 raft_consensus.cc:1565] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.417793 15569 raft_consensus.cc:1050] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:51.418174 15569 consensus_queue.cc:237] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:51.418378 12593 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.419958 14680 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.420313 12857 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.420861 15654 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.421303 15488 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.421558 15488 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.421684 12856 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.417029 12725 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.422425 15654 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.420279 15925 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.422860 12595 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.423995 12724 raft_consensus.cc:1203] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 19:56:51.420464 12723 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.421123 15472 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20240501 19:56:51.425510 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e4d79f5e11b4923acb2ce543884b54a with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.425604 15654 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.427160 16195 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.427165 12444 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 6 to 7, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.429075 15926 raft_consensus.cc:1050] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:51.429512 15926 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.430817 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4d79f5e11b4923acb2ce543884b54a with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:51.431329 14679 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.431402 12988 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.431402 13712 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.431818 13712 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.432082 13712 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.434937 12857 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.435421 15654 raft_consensus.cc:1050] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:51.435694 14284 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.435817 15654 consensus_queue.cc:237] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:51.436201 15553 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.436456 15553 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.437649 14678 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.437815 12725 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.437812 12988 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.438207 15568 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.438401 16123 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.438550 15569 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.438900 15569 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.440467 14678 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.441365 15500 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.441696 12595 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.442786 12856 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.443591 14283 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.444737 12725 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:51.445430 12444 catalog_manager.cc:5484] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:51.446509 12453 catalog_manager.cc:5484] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 6 to 7, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.446870 15542 raft_consensus.cc:1050] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:51.447260 15542 consensus_queue.cc:237] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.448134 14549 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:51.448652 15569 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.448773 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:51.449323 12988 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.449512 14417 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.449651 14680 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.450712 12725 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.450767 14680 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.451623 16064 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.451864 16267 tablet_replica.cc:330] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:51.451889 16064 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.451993 16267 raft_consensus.cc:2229] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.452098 16064 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.452443 12988 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.452780 12595 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.454159 14285 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.454573 13712 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.456017 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.456296 12987 consensus_queue.cc:237] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.456899 14285 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.457863 14286 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.458391 15508 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 19:56:51.458889 12923 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f08535cb2f46464b9e510a45fee67b4f. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.459168 12453 catalog_manager.cc:5484] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.459542 12452 catalog_manager.cc:5484] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.459632 14144 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.460131 12854 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.460891 12595 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.461272 15500 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.462261 15926 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.462702 14549 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.464010 15678 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.464190 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:51.465631 16179 tablet_replica.cc:330] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:51.465781 16179 raft_consensus.cc:2229] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.465934 16179 raft_consensus.cc:2258] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.466050 12593 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.466212 16262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544caa38514643d5b81cd6ed85e0bae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.466708 12452 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 8 to 9. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.467216 16179 ts_tablet_manager.cc:1875] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.468257 12988 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.469027 16262 tablet_copy_client.cc:783] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.469206 16262 tablet_copy_client.cc:647] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.470396 14680 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.470759 12855 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.471865 12452 catalog_manager.cc:5484] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.472746 16179 ts_tablet_manager.cc:1888] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:51.472811 16179 log.cc:1196] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/800d5f43cd1b4fb6bd8bbcac262aeb80
I20240501 19:56:51.473307 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 (table rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]) successfully deleted
I20240501 19:56:51.474993 15553 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.476331 12726 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.477216 12987 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.477648 12857 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.477660 14284 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.477949 15500 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.479950 12726 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.480315 12453 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.480710 12593 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:51.452133 16267 raft_consensus.cc:2258] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.481417 15926 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:51.481864 12857 consensus_queue.cc:237] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:51.481945 16267 ts_tablet_manager.cc:1875] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.482228 12593 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.483572 12595 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.483597 12987 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.483858 14680 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.484051 15542 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.484580 15542 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.484684 16064 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.485253 12723 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.487411 12593 consensus_queue.cc:237] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:51.487453 12594 consensus_queue.cc:237] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:51.487965 15542 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.489444 14678 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.490052 12594 consensus_queue.cc:237] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:51.490605 16267 ts_tablet_manager.cc:1888] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:51.490682 16267 log.cc:1196] T 7e4d79f5e11b4923acb2ce543884b54a P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/7e4d79f5e11b4923acb2ce543884b54a
I20240501 19:56:51.491109 12723 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.491463 15654 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.491529 15500 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.492130 14678 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.493619 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f08535cb2f46464b9e510a45fee67b4f with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.493991 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 7e4d79f5e11b4923acb2ce543884b54a (table rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]) successfully deleted
I20240501 19:56:51.494122 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fb481d394034a8f85af613beb5c9ec8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.494310 14680 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.494473 16123 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.494457 14467 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.494179 14678 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.494835 12725 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 19:56:51.494801 14284 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.495019 12857 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.496692 14467 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:56:51.496744 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): Couldn't send request to peer a238fc0c6d844cedb8adccb97d01ae5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8b61e94fc7e541c8aa84957560508a31. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.496858 12594 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.497175 14467 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.497444 15654 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.497710 12444 catalog_manager.cc:5484] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.498888 14548 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:51.498915 12452 catalog_manager.cc:5484] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 7 to 8, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.499127 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db240ec5f8b2444fb6114ad814f3cdb7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.499452 13712 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:51.500855 15827 ts_tablet_manager.cc:902] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:51.500836 12723 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.500824 12593 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.501341 15827 tablet_copy_client.cc:308] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:51.501726 12594 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:51.502060 12987 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.502157 12745 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 8bc9fe9ade8246779164d6db31b11e16 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:41689)
I20240501 19:56:51.502241 12745 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 8bc9fe9ade8246779164d6db31b11e16 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:41689: session id = 1d49e80be3d04fb99b711c62d61bac09-8bc9fe9ade8246779164d6db31b11e16
I20240501 19:56:51.502460 14467 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.503214 12745 tablet_copy_source_session.cc:205] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.503768 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb481d394034a8f85af613beb5c9ec8 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca06dad25a645bca71b09741798194a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51964
I20240501 19:56:51.503980 16178 tablet_replica.cc:330] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: stopping tablet replica
I20240501 19:56:51.504094 16178 raft_consensus.cc:2229] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Raft consensus shutting down.
I20240501 19:56:51.504192 16178 pending_rounds.cc:62] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Trying to abort 1 pending ops.
I20240501 19:56:51.504271 16178 pending_rounds.cc:68] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022974613286830080 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "0fb481d394034a8f85af613beb5c9ec8" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } } }
I20240501 19:56:51.504418 16178 raft_consensus.cc:2894] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 19:56:51.504479 16178 raft_consensus.cc:2866] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:51.504519 16178 raft_consensus.cc:2258] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Raft consensus is shut down!
I20240501 19:56:51.504403 14467 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.505451 12855 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.505666 16178 ts_tablet_manager.cc:1875] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.505874 12723 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.505998 15925 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.506192 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9fe9ade8246779164d6db31b11e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.507723 16178 ts_tablet_manager.cc:1888] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:51.507797 16178 log.cc:1196] T 0fb481d394034a8f85af613beb5c9ec8 P 2ca06dad25a645bca71b09741798194a: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/wals/0fb481d394034a8f85af613beb5c9ec8
I20240501 19:56:51.508280 15569 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.508409 15827 tablet_copy_client.cc:783] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.508553 15827 tablet_copy_client.cc:647] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.508881 12444 catalog_manager.cc:5484] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 5 to 6, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.509320 12988 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.510849 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 22b272a8582d4060a6a527ac4fce9ae5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.510993 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d940b16d07d54945abe4a10a11dd6a2a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.510977 15925 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:51.511224 12439 catalog_manager.cc:4830] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): tablet 0fb481d394034a8f85af613beb5c9ec8 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
I20240501 19:56:51.511402 14678 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.513751 12445 catalog_manager.cc:5484] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.514925 14548 raft_consensus.cc:1203] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:51.515066 14284 raft_consensus.cc:1203] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:51.516798 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet d940b16d07d54945abe4a10a11dd6a2a with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:51.517347 15926 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
I20240501 19:56:51.517849 12593 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
I20240501 19:56:51.518913 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8b61e94fc7e541c8aa84957560508a31 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 19:56:51.519693 12726 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.520641 12986 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.520794 14417 raft_consensus.cc:1203] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:51.522049 14549 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
I20240501 19:56:51.524492 12445 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
I20240501 19:56:51.526432 12857 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.528249 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet db240ec5f8b2444fb6114ad814f3cdb7 with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:51.530889 12987 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.532316 15925 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.532390 16267 tablet_replica.cc:330] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:51.532548 16267 raft_consensus.cc:2229] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.532716 16267 raft_consensus.cc:2258] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.533353 16323 tablet_replica.cc:330] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:51.533429 14529 tablet_service.cc:1514] Processing DeleteTablet for tablet f08535cb2f46464b9e510a45fee67b4f with delete_type TABLET_DATA_TOMBSTONED (TS 1d49e80be3d04fb99b711c62d61bac09 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:52744
I20240501 19:56:51.533418 12444 catalog_manager.cc:5484] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.533658 16323 raft_consensus.cc:2229] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.533872 16323 raft_consensus.cc:2258] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.534801 14680 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.535184 14285 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.535583 15925 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.536105 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet 22b272a8582d4060a6a527ac4fce9ae5 with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:51.537292 15926 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.538074 14284 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.539866 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e84a3865ce64eaebad4df46ec01842d with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.540279 12987 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.541229 12444 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:51.542027 12857 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.542878 16267 ts_tablet_manager.cc:1875] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.544268 12987 consensus_queue.cc:237] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.545359 14677 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.546353 12726 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:51.546826 15508 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:51.546967 15827 tablet_copy_client.cc:523] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.548592 14284 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:51.549129 15508 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:51.549476 12593 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:51.549839 16262 tablet_copy_client.cc:523] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.551133 16262 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:51.554725 16295 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:51.554790 15827 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:51.554741 14660 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e84a3865ce64eaebad4df46ec01842d with delete_type TABLET_DATA_TOMBSTONED (TS c6d2c7096e404ea9a6f73b123d74c622 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52566
I20240501 19:56:51.558531 12857 consensus_queue.cc:237] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:51.559676 16333 tablet_replica.cc:330] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: stopping tablet replica
I20240501 19:56:51.559866 16333 raft_consensus.cc:2229] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 19:56:51.560077 16333 pending_rounds.cc:62] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Trying to abort 1 pending ops.
I20240501 19:56:51.560186 16333 pending_rounds.cc:68] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7022974613146984448 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f08535cb2f46464b9e510a45fee67b4f" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } } }
I20240501 19:56:51.560551 16333 raft_consensus.cc:2894] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240501 19:56:51.560683 16333 raft_consensus.cc:2866] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:51.560735 14679 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.560798 16333 raft_consensus.cc:2258] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 19:56:51.560850 16295 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.561159 16064 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.561525 16333 ts_tablet_manager.cc:1875] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.562434 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f08535cb2f46464b9e510a45fee67b4f with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.562793 12593 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.562835 16064 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.563421 12593 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.564417 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fb481d394034a8f85af613beb5c9ec8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.564886 16066 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.565083 14679 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.567512 12452 catalog_manager.cc:5484] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.567634 16267 ts_tablet_manager.cc:1888] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:51.569134 16267 log.cc:1196] T db240ec5f8b2444fb6114ad814f3cdb7 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/db240ec5f8b2444fb6114ad814f3cdb7
I20240501 19:56:51.569447 12594 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:51.569834 12453 catalog_manager.cc:5484] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:51.570672 16343 tablet_replica.cc:330] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: stopping tablet replica
I20240501 19:56:51.570935 16328 ts_tablet_manager.cc:902] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:51.571359 14284 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.571708 16343 raft_consensus.cc:2229] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 19:56:51.571990 16343 pending_rounds.cc:62] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Trying to abort 1 pending ops.
W20240501 19:56:51.572180 12437 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 0fb481d394034a8f85af613beb5c9ec8 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER failed with leader a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 7 but the committed config has opid_index of 8
I20240501 19:56:51.572584 16328 tablet_copy_client.cc:308] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:51.572813 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb481d394034a8f85af613beb5c9ec8 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:51.572053 16343 pending_rounds.cc:68] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022974613214195712 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "3e84a3865ce64eaebad4df46ec01842d" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } } }
I20240501 19:56:51.572911 16343 raft_consensus.cc:2894] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 19:56:51.572991 16343 raft_consensus.cc:2866] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:51.573071 16343 raft_consensus.cc:2258] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 19:56:51.573143 16179 tablet_replica.cc:330] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:51.573274 16179 raft_consensus.cc:2229] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.573438 16179 raft_consensus.cc:2258] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.573598 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet f08535cb2f46464b9e510a45fee67b4f with delete_type TABLET_DATA_TOMBSTONED (TS 2ca06dad25a645bca71b09741798194a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51964
W20240501 19:56:51.574102 12439 catalog_manager.cc:4797] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): delete failed for tablet f08535cb2f46464b9e510a45fee67b4f because the tablet was not found. No further retry: Not found: Tablet not found: f08535cb2f46464b9e510a45fee67b4f
I20240501 19:56:51.575044 12987 consensus_queue.cc:237] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:51.576475 16179 ts_tablet_manager.cc:1875] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.576957 12593 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:51.577340 15508 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:51.578549 16179 ts_tablet_manager.cc:1888] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:51.578613 16179 log.cc:1196] T 0fb481d394034a8f85af613beb5c9ec8 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/0fb481d394034a8f85af613beb5c9ec8
I20240501 19:56:51.578713 14417 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.579253 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 0fb481d394034a8f85af613beb5c9ec8 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
I20240501 19:56:51.579978 16333 ts_tablet_manager.cc:1888] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:51.580040 16333 log.cc:1196] T f08535cb2f46464b9e510a45fee67b4f P 1d49e80be3d04fb99b711c62d61bac09: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-6/wal/wals/f08535cb2f46464b9e510a45fee67b4f
I20240501 19:56:51.580389 12987 consensus_queue.cc:237] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.581030 12440 catalog_manager.cc:4830] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): tablet f08535cb2f46464b9e510a45fee67b4f (table rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]) successfully deleted
I20240501 19:56:51.581533 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet 955edb7c12fa4b32ac58827f131931e3 from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:42155)
I20240501 19:56:51.581617 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet 955edb7c12fa4b32ac58827f131931e3 from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:42155: session id = c6d2c7096e404ea9a6f73b123d74c622-955edb7c12fa4b32ac58827f131931e3
I20240501 19:56:51.582449 12614 tablet_copy_source_session.cc:205] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.584151 14285 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:51.584566 15508 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:51.586290 15508 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.586777 16328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955edb7c12fa4b32ac58827f131931e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.587203 14284 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.587945 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f08535cb2f46464b9e510a45fee67b4f with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.589187 16328 tablet_copy_client.cc:783] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.589433 12444 catalog_manager.cc:5484] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 9 to 10, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:51.589588 16328 tablet_copy_client.cc:647] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
W20240501 19:56:51.591606 12923 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce6add2f19a649b3a12497951047705c. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.592260 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet db240ec5f8b2444fb6114ad814f3cdb7 (table rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]) successfully deleted
I20240501 19:56:51.592293 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet f08535cb2f46464b9e510a45fee67b4f with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:51.592516 16072 tablet_replica.cc:330] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:51.592636 16072 raft_consensus.cc:2229] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.585796 12857 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:51.592854 16072 raft_consensus.cc:2258] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.586483 16066 raft_consensus.cc:1050] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:51.593396 16066 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.593870 16343 ts_tablet_manager.cc:1875] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.595081 14284 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.595080 12987 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.595433 16064 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.595800 16064 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.595860 16072 ts_tablet_manager.cc:1875] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.571177 12723 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.597931 16072 ts_tablet_manager.cc:1888] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 19:56:51.597994 16072 log.cc:1196] T f08535cb2f46464b9e510a45fee67b4f P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/f08535cb2f46464b9e510a45fee67b4f
I20240501 19:56:51.594933 12725 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.598487 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet f08535cb2f46464b9e510a45fee67b4f (table rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]) successfully deleted
I20240501 19:56:51.598572 16343 ts_tablet_manager.cc:1888] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:51.598649 16343 log.cc:1196] T 3e84a3865ce64eaebad4df46ec01842d P c6d2c7096e404ea9a6f73b123d74c622: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-7/wal/wals/3e84a3865ce64eaebad4df46ec01842d
I20240501 19:56:51.598734 16064 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.599393 12439 catalog_manager.cc:4830] TS c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): tablet 3e84a3865ce64eaebad4df46ec01842d (table rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]) successfully deleted
I20240501 19:56:51.600111 15508 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:51.602157 12726 raft_consensus.cc:1261] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:51.567286 16323 ts_tablet_manager.cc:1875] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.602743 16295 consensus_queue.cc:1035] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:51.603905 12723 raft_consensus.cc:2932] T f08535cb2f46464b9e510a45fee67b4f P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.604785 16328 tablet_copy_client.cc:523] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.606963 16328 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:51.606675 16323 ts_tablet_manager.cc:1888] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:51.607167 16323 log.cc:1196] T d940b16d07d54945abe4a10a11dd6a2a P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/d940b16d07d54945abe4a10a11dd6a2a
I20240501 19:56:51.607759 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet d940b16d07d54945abe4a10a11dd6a2a (table rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]) successfully deleted
I20240501 19:56:51.608023 16323 tablet_replica.cc:330] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:51.608173 16323 raft_consensus.cc:2229] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.608316 16323 raft_consensus.cc:2258] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.610757 16295 raft_consensus.cc:1050] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:51.611166 12594 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:51.611166 16295 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:51.612718 14416 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.612895 12593 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.612895 12857 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.612902 14549 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.613118 14467 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.613243 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.613327 14416 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.613350 15488 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.613636 15488 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.614735 16064 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.615423 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.615442 12723 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.615792 14467 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.616515 12987 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.616736 14285 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.616888 16123 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:56:51.618713 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d7ea564ea504163a1e17561be9a5130. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.619417 12986 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.619503 15488 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.619588 15508 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:51.621348 12593 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:51.622740 12450 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.624161 12444 catalog_manager.cc:5484] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:51.622037 12857 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:51.629464 12725 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.634521 12857 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:51.636159 14284 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.636694 16065 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.637343 12725 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.640419 16065 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.642390 16065 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.645201 14284 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.647748 12444 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.648054 15827 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.648345 15827 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:51.648533 15827 ts_tablet_manager.cc:1367] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.094s user 0.027s sys 0.000s
I20240501 19:56:51.649019 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04ccc8addc55441b9ab9d8265b90b03b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.649030 15827 raft_consensus.cc:348] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:51.649188 15827 raft_consensus.cc:729] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:51.649302 15827 consensus_queue.cc:260] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:51.649854 12725 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:51.651155 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ccc8addc55441b9ab9d8265b90b03b with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:51.651229 16323 ts_tablet_manager.cc:1875] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.651366 16179 tablet_replica.cc:330] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:51.651546 16179 raft_consensus.cc:2229] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.651731 16179 raft_consensus.cc:2258] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.652472 15827 ts_tablet_manager.cc:1398] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 19:56:51.652506 16179 ts_tablet_manager.cc:1875] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.654700 16179 ts_tablet_manager.cc:1888] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:51.654906 16179 log.cc:1196] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/04ccc8addc55441b9ab9d8265b90b03b
I20240501 19:56:51.655598 12745 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-8bc9fe9ade8246779164d6db31b11e16 received from {username='slave'} at 127.5.139.71:41689
I20240501 19:56:51.655673 12745 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-8bc9fe9ade8246779164d6db31b11e16 on tablet 8bc9fe9ade8246779164d6db31b11e16 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:51.655772 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 04ccc8addc55441b9ab9d8265b90b03b (table rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]) successfully deleted
I20240501 19:56:51.658003 16323 ts_tablet_manager.cc:1888] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:51.658083 16323 log.cc:1196] T 22b272a8582d4060a6a527ac4fce9ae5 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/22b272a8582d4060a6a527ac4fce9ae5
I20240501 19:56:51.659715 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet 22b272a8582d4060a6a527ac4fce9ae5 (table rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]) successfully deleted
I20240501 19:56:51.663094 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.665706 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:51.673481 12855 consensus_queue.cc:237] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:51.675750 12725 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:51.675981 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bf64d9f3ce94db28b032cdcd795a255. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.676172 15542 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.676596 12986 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.677104 15542 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.678745 15542 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.681398 12725 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
W20240501 19:56:51.681576 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bf64d9f3ce94db28b032cdcd795a255. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.683509 12986 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.684334 12449 catalog_manager.cc:5484] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 4 to 5, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.689500 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:51.697094 16123 raft_consensus.cc:1013] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: attempt to promote peer 2ca06dad25a645bca71b09741798194a: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 19:56:51.698235 15654 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:51.699913 16196 raft_consensus.cc:1013] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: attempt to promote peer 1d49e80be3d04fb99b711c62d61bac09: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 19:56:51.701843 12723 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:51.704555 12449 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.705605 14416 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:51.708484 16123 raft_consensus.cc:1050] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:51.708957 16123 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
W20240501 19:56:51.710786 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d7ea564ea504163a1e17561be9a5130. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.713022 14416 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.713549 12723 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.713685 14467 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.714424 14467 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.747148 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.753692 15654 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:51.757227 12448 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.759047 14416 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:51.763382 12594 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:51.765383 12723 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
W20240501 19:56:51.767186 12530 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf6948589837482a9f580731e3b861b8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.767640 16296 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
W20240501 19:56:51.768010 12923 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.5.139.70:33933, user_credentials={real_user=slave}} blocked reactor thread for 50281.5us
I20240501 19:56:51.768361 12855 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.768922 12986 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.769629 12988 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.770476 14284 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.770982 12855 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.771865 14416 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
W20240501 19:56:51.773152 16262 log.cc:924] Time spent T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: Append to log took a long time: real 0.168s user 0.004s sys 0.000s
I20240501 19:56:51.773548 14549 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.773607 12593 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.774432 12452 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:51.774981 14467 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.775280 14467 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.776563 15568 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.777449 15568 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.778594 12988 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:51.779058 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.779394 15488 raft_consensus.cc:1050] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:51.779843 15488 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:51.779990 12988 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:51.781349 12857 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.781639 12594 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.781956 14416 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.782092 16196 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.782241 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.783282 14549 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.784099 16196 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.784715 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.785029 16296 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
W20240501 19:56:51.787926 12440 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 851eeb0ab3364529907d9d09f2a4022e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240501 19:56:51.788437 15827 ts_tablet_manager.cc:902] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:51.788533 12594 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.788832 12857 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.791121 12448 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.792083 14416 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.792539 15827 tablet_copy_client.cc:308] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:51.793310 12745 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet e4b15761177f4cffb7aa5e4d16e42ac7 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:41689)
I20240501 19:56:51.793426 12745 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet e4b15761177f4cffb7aa5e4d16e42ac7 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:41689: session id = 1d49e80be3d04fb99b711c62d61bac09-e4b15761177f4cffb7aa5e4d16e42ac7
I20240501 19:56:51.794428 12745 tablet_copy_source_session.cc:205] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:51.795305 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b15761177f4cffb7aa5e4d16e42ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:51.797292 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.797663 15827 tablet_copy_client.cc:783] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:51.797818 15827 tablet_copy_client.cc:647] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:51.799965 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:51.801813 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.824766 15827 tablet_copy_client.cc:523] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:51.826197 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.826508 15827 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:51.827301 12723 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
W20240501 19:56:51.829106 12662 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.829685 12988 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.830094 15553 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.833288 15553 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.835259 12988 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.836894 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4b15761177f4cffb7aa5e4d16e42ac7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.837359 14416 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.837711 15925 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.839231 12450 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 6 to 7, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.840478 14416 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:51.841900 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b15761177f4cffb7aa5e4d16e42ac7 with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:51.842108 16267 tablet_replica.cc:330] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:51.842247 16267 raft_consensus.cc:2229] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.842451 16267 raft_consensus.cc:2258] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.842615 14549 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.843304 16267 ts_tablet_manager.cc:1875] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.846004 16267 ts_tablet_manager.cc:1888] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:51.846078 16267 log.cc:1196] T e4b15761177f4cffb7aa5e4d16e42ac7 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/e4b15761177f4cffb7aa5e4d16e42ac7
I20240501 19:56:51.846760 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet e4b15761177f4cffb7aa5e4d16e42ac7 (table rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]) successfully deleted
I20240501 19:56:51.852245 12988 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:51.853896 12857 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.854318 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.855019 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:51.857976 14549 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.858376 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.859128 14416 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:51.859479 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:51.860991 15508 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.863062 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 851eeb0ab3364529907d9d09f2a4022e with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.864740 12857 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.865695 12448 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.867009 14549 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.869496 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet 851eeb0ab3364529907d9d09f2a4022e with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:51.869484 12988 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:51.869710 16072 tablet_replica.cc:330] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:51.869835 16072 raft_consensus.cc:2229] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.869999 16072 raft_consensus.cc:2258] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.870693 16072 ts_tablet_manager.cc:1875] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.871644 12857 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:51.872027 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:51.872833 16072 ts_tablet_manager.cc:1888] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:51.872895 16072 log.cc:1196] T 851eeb0ab3364529907d9d09f2a4022e P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/851eeb0ab3364529907d9d09f2a4022e
I20240501 19:56:51.873333 14549 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:51.873706 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:51.874384 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 851eeb0ab3364529907d9d09f2a4022e (table rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]) successfully deleted
I20240501 19:56:51.875488 16296 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.876091 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:51.876142 16262 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.868425 14416 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:51.877233 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 851eeb0ab3364529907d9d09f2a4022e with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.877600 16262 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:51.878839 12857 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.879266 12452 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.881755 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 851eeb0ab3364529907d9d09f2a4022e with delete_type TABLET_DATA_TOMBSTONED (TS 2ca06dad25a645bca71b09741798194a not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51964
I20240501 19:56:51.881983 16178 tablet_replica.cc:330] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: stopping tablet replica
I20240501 19:56:51.882109 16178 raft_consensus.cc:2229] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.882242 16178 raft_consensus.cc:2258] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.883028 16178 ts_tablet_manager.cc:1875] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.882907 14549 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:51.885241 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.885705 16178 ts_tablet_manager.cc:1888] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:51.885783 16178 log.cc:1196] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/wals/851eeb0ab3364529907d9d09f2a4022e
I20240501 19:56:51.886958 12439 catalog_manager.cc:4830] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): tablet 851eeb0ab3364529907d9d09f2a4022e (table rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]) successfully deleted
I20240501 19:56:51.901347 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:51.903781 15568 raft_consensus.cc:1013] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: attempt to promote peer 9069ec6fcdb248479037f2d37e0e851d: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 19:56:51.905650 14467 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.908113 14284 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.909220 12452 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.911197 12855 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.913874 16123 raft_consensus.cc:1050] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:51.914290 16123 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
W20240501 19:56:51.915982 12530 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf6948589837482a9f580731e3b861b8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.916471 12855 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.916920 15568 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.917630 14284 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.918056 15568 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:51.922055 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:51.907362 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:51.929411 16262 ts_tablet_manager.cc:1367] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.378s user 0.026s sys 0.000s
I20240501 19:56:51.929878 16262 raft_consensus.cc:348] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:51.929999 16262 raft_consensus.cc:729] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:51.930087 16262 consensus_queue.cc:260] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:51.930351 16262 ts_tablet_manager.cc:1398] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:51.930970 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-544caa38514643d5b81cd6ed85e0bae7 received from {username='slave'} at 127.5.139.72:42155
I20240501 19:56:51.931043 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-544caa38514643d5b81cd6ed85e0bae7 on tablet 544caa38514643d5b81cd6ed85e0bae7 with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:51.938519 16328 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:51.938650 12988 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:51.939163 14467 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.939901 12988 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.944273 14467 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.946518 16328 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:51.946817 12988 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.946969 12453 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.947304 16328 ts_tablet_manager.cc:1367] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.340s user 0.026s sys 0.000s
I20240501 19:56:51.948722 12855 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.949451 16328 raft_consensus.cc:348] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:51.953897 16328 raft_consensus.cc:729] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:51.954087 16328 consensus_queue.cc:260] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:51.954484 16328 ts_tablet_manager.cc:1398] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240501 19:56:51.955093 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-955edb7c12fa4b32ac58827f131931e3 received from {username='slave'} at 127.5.139.72:42155
I20240501 19:56:51.955206 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-955edb7c12fa4b32ac58827f131931e3 on tablet 955edb7c12fa4b32ac58827f131931e3 with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:51.960737 14284 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.962513 12594 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
W20240501 19:56:51.965198 12530 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cf6948589837482a9f580731e3b861b8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:51.967612 14284 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.968072 12855 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:51.968477 15568 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.970023 15654 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:51.973292 14467 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.975222 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf6948589837482a9f580731e3b861b8 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:51.976154 14284 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.976377 12453 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 6 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:51.979583 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6948589837482a9f580731e3b861b8 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:51.979933 16179 tablet_replica.cc:330] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:51.980070 16179 raft_consensus.cc:2229] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:51.980232 16179 raft_consensus.cc:2258] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:51.982486 16328 ts_tablet_manager.cc:902] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:51.982860 16179 ts_tablet_manager.cc:1875] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:51.982947 16328 tablet_copy_client.cc:308] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:51.985531 16179 ts_tablet_manager.cc:1888] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:51.985689 16179 log.cc:1196] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/cf6948589837482a9f580731e3b861b8
I20240501 19:56:51.986505 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet cf6948589837482a9f580731e3b861b8 (table rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]) successfully deleted
I20240501 19:56:51.988742 12855 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:51.992178 15926 raft_consensus.cc:1050] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:51.992615 15926 consensus_queue.cc:237] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:51.994974 12594 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:51.995350 15925 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:51.997433 15827 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:52.000245 12855 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.000638 15925 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.001775 14549 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.002132 15925 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.004500 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.006786 15827 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:52.008045 15827 ts_tablet_manager.cc:1367] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.182s user 0.029s sys 0.000s
I20240501 19:56:52.008698 15827 raft_consensus.cc:348] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.008867 15827 raft_consensus.cc:729] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:52.008965 15827 consensus_queue.cc:260] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.014833 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.019806 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet ce6add2f19a649b3a12497951047705c from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:44431)
I20240501 19:56:52.019924 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet ce6add2f19a649b3a12497951047705c from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:44431: session id = c6d2c7096e404ea9a6f73b123d74c622-ce6add2f19a649b3a12497951047705c
I20240501 19:56:52.020884 13007 tablet_copy_source_session.cc:205] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.023896 16328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce6add2f19a649b3a12497951047705c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.026332 16328 tablet_copy_client.cc:783] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.026502 16328 tablet_copy_client.cc:647] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.029095 16328 tablet_copy_client.cc:523] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:52.030561 16328 tablet_bootstrap.cc:492] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:52.032397 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.037163 15827 ts_tablet_manager.cc:1398] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.029s user 0.000s sys 0.001s
I20240501 19:56:52.037689 12745 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-e4b15761177f4cffb7aa5e4d16e42ac7 received from {username='slave'} at 127.5.139.71:41689
I20240501 19:56:52.038017 12745 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-e4b15761177f4cffb7aa5e4d16e42ac7 on tablet e4b15761177f4cffb7aa5e4d16e42ac7 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:52.049656 15827 ts_tablet_manager.cc:902] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:52.049644 13712 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.050132 15827 tablet_copy_client.cc:308] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:52.052731 14548 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.052968 12452 catalog_manager.cc:5484] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:52.055286 12745 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 7e4d79f5e11b4923acb2ce543884b54a from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:41689)
I20240501 19:56:52.055394 12745 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 7e4d79f5e11b4923acb2ce543884b54a from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:41689: session id = 1d49e80be3d04fb99b711c62d61bac09-7e4d79f5e11b4923acb2ce543884b54a
I20240501 19:56:52.055687 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:52.056268 12745 tablet_copy_source_session.cc:205] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.056826 12857 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.057147 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4d79f5e11b4923acb2ce543884b54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.059322 15827 tablet_copy_client.cc:783] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.059473 15827 tablet_copy_client.cc:647] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.063413 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.063884 15827 tablet_copy_client.cc:523] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:52.065212 15827 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:52.073750 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.077601 16328 tablet_bootstrap.cc:492] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:52.077885 16328 tablet_bootstrap.cc:492] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:52.078054 16328 ts_tablet_manager.cc:1367] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.048s user 0.024s sys 0.000s
I20240501 19:56:52.078581 16328 raft_consensus.cc:348] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.078718 16328 raft_consensus.cc:729] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:52.078819 16328 consensus_queue.cc:260] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.079075 16328 ts_tablet_manager.cc:1398] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.080368 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-ce6add2f19a649b3a12497951047705c received from {username='slave'} at 127.5.139.72:44431
I20240501 19:56:52.080458 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-ce6add2f19a649b3a12497951047705c on tablet ce6add2f19a649b3a12497951047705c with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:52.081353 14548 raft_consensus.cc:1203] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:52.084498 16510 ts_tablet_manager.cc:902] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:52.085289 16510 tablet_copy_client.cc:308] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:52.085630 12988 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.085867 12745 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 8b61e94fc7e541c8aa84957560508a31 from peer a238fc0c6d844cedb8adccb97d01ae5e ({username='slave'} at 127.5.139.67:58333)
I20240501 19:56:52.085975 12745 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 8b61e94fc7e541c8aa84957560508a31 from peer a238fc0c6d844cedb8adccb97d01ae5e at {username='slave'} at 127.5.139.67:58333: session id = a238fc0c6d844cedb8adccb97d01ae5e-8b61e94fc7e541c8aa84957560508a31
I20240501 19:56:52.086836 12745 tablet_copy_source_session.cc:205] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.087500 12723 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 19:56:52.087745 12923 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 745e39ec23db4f90bb8b1e8b49a50244. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.088279 16196 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.088440 16510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b61e94fc7e541c8aa84957560508a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.090003 16196 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:52.093245 12723 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:52.095759 12594 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.104238 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.104568 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.105556 15827 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:52.105967 15827 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:52.106170 15827 ts_tablet_manager.cc:1367] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.041s user 0.021s sys 0.007s
I20240501 19:56:52.106506 12452 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.106743 15827 raft_consensus.cc:348] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.106907 15827 raft_consensus.cc:729] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:52.107010 15827 consensus_queue.cc:260] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.107271 15827 ts_tablet_manager.cc:1398] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.107820 12745 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-7e4d79f5e11b4923acb2ce543884b54a received from {username='slave'} at 127.5.139.71:41689
I20240501 19:56:52.107901 12745 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-7e4d79f5e11b4923acb2ce543884b54a on tablet 7e4d79f5e11b4923acb2ce543884b54a with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:52.114570 16510 tablet_copy_client.cc:783] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.114825 16510 tablet_copy_client.cc:647] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.119599 14679 raft_consensus.cc:1203] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:52.122822 16510 tablet_copy_client.cc:523] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:52.123142 12723 consensus_queue.cc:237] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:52.125101 14548 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.125814 15553 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.126015 16510 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 19:56:52.127739 15553 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.129446 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 32a17c2e392a49b7b181fa7a25a778a1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.130209 14548 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.130468 12594 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.130877 15926 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.133822 12594 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.135324 12453 catalog_manager.cc:5484] T 32a17c2e392a49b7b181fa7a25a778a1 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 5 to 6, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.136472 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.141942 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a17c2e392a49b7b181fa7a25a778a1 with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:52.142150 16267 tablet_replica.cc:330] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:52.142257 16267 raft_consensus.cc:2229] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.142400 16267 raft_consensus.cc:2258] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.143553 16267 ts_tablet_manager.cc:1875] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.145713 16267 ts_tablet_manager.cc:1888] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:52.145773 16267 log.cc:1196] T 32a17c2e392a49b7b181fa7a25a778a1 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/32a17c2e392a49b7b181fa7a25a778a1
I20240501 19:56:52.146490 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 32a17c2e392a49b7b181fa7a25a778a1 (table rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]) successfully deleted
I20240501 19:56:52.150564 15925 raft_consensus.cc:1050] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:52.150951 15925 consensus_queue.cc:237] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:52.153502 14416 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.153549 12594 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.154033 15553 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.154659 15553 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.156359 12988 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.156764 15553 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.157316 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.171893 14284 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.172467 16296 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.174257 14284 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:52.178663 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.183454 15827 ts_tablet_manager.cc:902] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:52.183892 15827 tablet_copy_client.cc:308] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:52.184870 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 0bf64d9f3ce94db28b032cdcd795a255 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:50211)
I20240501 19:56:52.184978 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 0bf64d9f3ce94db28b032cdcd795a255 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:50211: session id = 1d49e80be3d04fb99b711c62d61bac09-0bf64d9f3ce94db28b032cdcd795a255
I20240501 19:56:52.185608 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:52.186005 12877 tablet_copy_source_session.cc:205] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.186887 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf64d9f3ce94db28b032cdcd795a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.192467 15827 tablet_copy_client.cc:783] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.192638 15827 tablet_copy_client.cc:647] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.192831 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.196199 15926 raft_consensus.cc:1050] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:52.196594 15926 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.198064 14548 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.198632 15472 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.200054 12988 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.200414 15039 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.200472 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.202410 15039 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:52.203248 14416 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.204445 12986 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:52.207012 16510 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:52.207340 16510 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap complete.
I20240501 19:56:52.207291 12452 catalog_manager.cc:5484] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:52.207502 16510 ts_tablet_manager.cc:1367] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.083s user 0.025s sys 0.005s
I20240501 19:56:52.207927 16510 raft_consensus.cc:348] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 19:56:52.208043 16510 raft_consensus.cc:729] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: LEARNER
I20240501 19:56:52.208145 16510 consensus_queue.cc:260] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 19:56:52.208983 16510 ts_tablet_manager.cc:1398] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 19:56:52.209990 15926 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.210525 12745 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session a238fc0c6d844cedb8adccb97d01ae5e-8b61e94fc7e541c8aa84957560508a31 received from {username='slave'} at 127.5.139.67:58333
I20240501 19:56:52.210597 12745 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session a238fc0c6d844cedb8adccb97d01ae5e-8b61e94fc7e541c8aa84957560508a31 on tablet 8b61e94fc7e541c8aa84957560508a31 with peer a238fc0c6d844cedb8adccb97d01ae5e
I20240501 19:56:52.211658 12594 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:52.213549 14416 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:52.214342 12593 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.216506 13712 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.218581 12725 consensus_queue.cc:237] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:52.219741 12986 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.220160 15926 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.220232 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.221204 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.222321 14416 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.222726 13712 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.229256 15827 tablet_copy_client.cc:523] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:52.230518 15926 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:52.230793 15827 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:52.231060 14416 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:52.232676 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.234053 12986 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:52.234112 12449 catalog_manager.cc:5484] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a reported cstate change: config changed from index 5 to 6, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:52.239224 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a50be843fb49ee8a66c9a17d271d61 with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:52.239464 16072 tablet_replica.cc:330] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:52.239609 16072 raft_consensus.cc:2229] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.239771 16072 raft_consensus.cc:2258] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.240603 16072 ts_tablet_manager.cc:1875] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.242868 16072 ts_tablet_manager.cc:1888] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:52.242935 16072 log.cc:1196] T 12a50be843fb49ee8a66c9a17d271d61 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/12a50be843fb49ee8a66c9a17d271d61
I20240501 19:56:52.243625 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 12a50be843fb49ee8a66c9a17d271d61 (table rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]) successfully deleted
I20240501 19:56:52.253983 15926 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.256911 15553 raft_consensus.cc:1050] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:52.257236 15553 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:52.259979 14549 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.260075 12988 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.260146 14548 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.262435 13712 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.262683 12449 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.264825 13712 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.266027 14679 raft_consensus.cc:1203] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:52.265941 12988 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.267402 16328 ts_tablet_manager.cc:902] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:52.267879 16328 tablet_copy_client.cc:308] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:52.268347 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 0bf64d9f3ce94db28b032cdcd795a255 from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:37741)
I20240501 19:56:52.268445 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 0bf64d9f3ce94db28b032cdcd795a255 from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:37741: session id = c6d2c7096e404ea9a6f73b123d74c622-0bf64d9f3ce94db28b032cdcd795a255
I20240501 19:56:52.269389 12877 tablet_copy_source_session.cc:205] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.270224 16328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bf64d9f3ce94db28b032cdcd795a255. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.270488 14416 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.270704 12594 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.271162 15926 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.271328 12593 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.272645 14415 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.272827 16328 tablet_copy_client.cc:783] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.272970 16328 tablet_copy_client.cc:647] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.274180 15926 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.276854 12857 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:52.278515 12725 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.278890 16064 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.279229 12595 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:52.279647 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 557994658e2c4c8bbc1eeacdab958ddd. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.279834 16064 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.280246 16328 tablet_copy_client.cc:523] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 19:56:52.281338 12437 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 8bc9fe9ade8246779164d6db31b11e16 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 6 but the committed config has opid_index of 7
I20240501 19:56:52.281718 16328 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:52.282879 16065 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.283329 12725 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.284897 12595 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.285252 12594 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:52.287068 14548 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.288388 12450 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.290485 12453 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.292219 12988 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.293704 15039 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.294976 14415 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.295516 14548 raft_consensus.cc:1203] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:52.301008 12725 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.301595 14415 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.301615 15568 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:56:52.302276 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d7ea564ea504163a1e17561be9a5130. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.302428 15568 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.304967 15654 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.306895 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d7ea564ea504163a1e17561be9a5130 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.307716 14415 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.309921 12725 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.310068 12450 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a reported cstate change: config changed from index 6 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.313401 12857 consensus_queue.cc:237] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:52.314270 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7ea564ea504163a1e17561be9a5130 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:52.314489 16179 tablet_replica.cc:330] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:52.314621 16179 raft_consensus.cc:2229] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.314702 12725 consensus_queue.cc:237] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:52.314790 16179 raft_consensus.cc:2258] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.314846 14679 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:52.315250 16064 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:52.316184 16179 ts_tablet_manager.cc:1875] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.317229 12857 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.317847 15039 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.318488 16179 ts_tablet_manager.cc:1888] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:52.318568 16179 log.cc:1196] T 7d7ea564ea504163a1e17561be9a5130 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/7d7ea564ea504163a1e17561be9a5130
W20240501 19:56:52.319123 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61bc22cdc6ab45529066fe6f4d14d4c7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.319110 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 7d7ea564ea504163a1e17561be9a5130 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
W20240501 19:56:52.320736 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0fb481d394034a8f85af613beb5c9ec8. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.321471 16064 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.324419 14677 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.325753 15827 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:52.326014 15827 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:52.327317 15827 ts_tablet_manager.cc:1367] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.097s user 0.030s sys 0.001s
I20240501 19:56:52.327862 15827 raft_consensus.cc:348] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.328002 15827 raft_consensus.cc:729] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:52.328100 15827 consensus_queue.cc:260] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.328171 12595 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:52.328155 12450 catalog_manager.cc:5484] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 8 to 9, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.328805 15827 ts_tablet_manager.cc:1398] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.329142 16065 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:52.329237 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-0bf64d9f3ce94db28b032cdcd795a255 received from {username='slave'} at 127.5.139.71:50211
I20240501 19:56:52.329291 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-0bf64d9f3ce94db28b032cdcd795a255 on tablet 0bf64d9f3ce94db28b032cdcd795a255 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:52.329715 16328 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:52.330083 16328 tablet_bootstrap.cc:492] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:52.330298 16328 ts_tablet_manager.cc:1367] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.049s user 0.025s sys 0.004s
I20240501 19:56:52.330833 16328 raft_consensus.cc:348] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.330940 16328 raft_consensus.cc:729] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:52.331110 16328 consensus_queue.cc:260] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.331367 16328 ts_tablet_manager.cc:1398] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.331779 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-0bf64d9f3ce94db28b032cdcd795a255 received from {username='slave'} at 127.5.139.72:37741
I20240501 19:56:52.331853 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-0bf64d9f3ce94db28b032cdcd795a255 on tablet 0bf64d9f3ce94db28b032cdcd795a255 with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:52.333757 12595 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.336233 12595 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:52.337829 12593 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.338537 14679 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.338812 12857 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:52.338824 12530 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 955edb7c12fa4b32ac58827f131931e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.338990 15568 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.339031 12593 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.339895 15926 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.341915 13712 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.342221 14467 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.342375 12592 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.343847 12855 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.346307 12593 consensus_queue.cc:237] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.346753 15568 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.348402 12857 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.348841 14677 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.349839 15568 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.350618 14548 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.350870 12450 catalog_manager.cc:5484] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.352572 12855 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.353127 15568 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.354633 12725 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:52.354911 14467 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.356101 14548 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.356490 13712 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.356853 12855 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.357111 14415 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.357805 12593 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.359284 12988 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.359265 13712 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.359560 13712 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.359694 15569 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.360845 12986 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.362463 13712 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.362897 14677 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.364893 14548 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.365886 12449 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.367031 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8bc9fe9ade8246779164d6db31b11e16 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.367406 12593 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.367408 14415 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.369598 12452 catalog_manager.cc:5484] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.371012 12445 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 7 to 8, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:52.373721 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc9fe9ade8246779164d6db31b11e16 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:52.373715 12725 consensus_queue.cc:237] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.374015 12723 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:52.374922 16179 tablet_replica.cc:330] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:52.375097 16179 raft_consensus.cc:2229] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.375340 16179 raft_consensus.cc:2258] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.375943 14549 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 19:56:52.376060 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12a50be843fb49ee8a66c9a17d271d61. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.376475 15472 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:52.376524 14415 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.376801 14416 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:52.377120 14677 raft_consensus.cc:1203] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:52.377524 16179 ts_tablet_manager.cc:1875] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.377701 15553 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.377701 15472 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:52.378964 15925 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.380134 16179 ts_tablet_manager.cc:1888] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:52.380004 14549 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.380280 16179 log.cc:1196] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/8bc9fe9ade8246779164d6db31b11e16
I20240501 19:56:52.380836 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 8bc9fe9ade8246779164d6db31b11e16 (table rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]) successfully deleted
I20240501 19:56:52.381379 12988 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.381789 15925 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.382535 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8bc9fe9ade8246779164d6db31b11e16 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.382655 15925 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.384774 12445 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 8 to 9, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:52.387917 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bc9fe9ade8246779164d6db31b11e16 with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:52.388113 16072 tablet_replica.cc:330] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:52.388257 16072 raft_consensus.cc:2229] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.388437 16072 raft_consensus.cc:2258] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.386873 12988 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.389884 16072 ts_tablet_manager.cc:1875] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.392123 16072 ts_tablet_manager.cc:1888] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:52.392207 16072 log.cc:1196] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/8bc9fe9ade8246779164d6db31b11e16
I20240501 19:56:52.392316 12452 catalog_manager.cc:5484] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.393024 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 8bc9fe9ade8246779164d6db31b11e16 (table rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]) successfully deleted
I20240501 19:56:52.393276 14415 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.393296 14416 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
W20240501 19:56:52.398783 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d940b16d07d54945abe4a10a11dd6a2a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.399199 14467 raft_consensus.cc:1050] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:52.399629 14467 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:52.400131 12723 consensus_queue.cc:237] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.400069 14679 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.401486 12855 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.401610 14284 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.401964 12857 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.402074 16123 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:56:52.402784 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 800d5f43cd1b4fb6bd8bbcac262aeb80. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.402958 15472 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.404554 14679 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.404717 12988 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.404981 15472 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.405599 16123 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 19:56:52.406769 12530 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 955edb7c12fa4b32ac58827f131931e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.406881 15569 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.407406 15553 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.407743 14284 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.409759 15569 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.409973 12450 catalog_manager.cc:5484] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.412000 12855 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.412741 12857 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.414391 12988 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.414980 12450 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.424721 14679 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.426071 12593 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:52.427698 12988 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.427843 12857 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.428150 15654 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.428315 16123 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.430084 14467 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.432049 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 955edb7c12fa4b32ac58827f131931e3 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.433377 12988 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.435920 12451 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.435986 12857 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.439363 14679 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.439901 16123 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.440781 12855 consensus_queue.cc:237] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.440996 12854 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.441177 14679 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.443635 12725 raft_consensus.cc:1261] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.443635 12723 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.444384 14284 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:52.445295 16654 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.445335 16655 consensus_queue.cc:1035] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.445629 16654 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:52.445960 12857 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:52.447125 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 955edb7c12fa4b32ac58827f131931e3 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca06dad25a645bca71b09741798194a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51964
I20240501 19:56:52.447575 12726 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:52.447605 12790 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 557994658e2c4c8bbc1eeacdab958ddd. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.447906 16654 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.448089 12593 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.448751 16655 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
W20240501 19:56:52.448899 12792 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04ccc8addc55441b9ab9d8265b90b03b. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:52.448985 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d75bd8f836b947248ee124f456c7d721. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.449111 16065 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.449488 16064 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
W20240501 19:56:52.449095 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 557994658e2c4c8bbc1eeacdab958ddd. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.449684 14284 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:52.452005 12445 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.452603 12726 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:52.453253 12988 raft_consensus.cc:1261] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.454645 14144 consensus_queue.cc:1035] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.455164 12725 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.457564 12593 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.460120 16065 raft_consensus.cc:2932] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:52.460131 12451 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 4 to 5, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
W20240501 19:56:52.462493 12439 catalog_manager.cc:4797] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): delete failed for tablet 955edb7c12fa4b32ac58827f131931e3 because the tablet was not found. No further retry: Not found: Tablet not found: 955edb7c12fa4b32ac58827f131931e3
I20240501 19:56:52.462980 12723 raft_consensus.cc:2932] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:52.467412 12445 catalog_manager.cc:5484] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:52.475831 12988 raft_consensus.cc:2932] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:52.483691 12988 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:52.485193 14284 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:52.485723 16296 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 19:56:52.486951 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.487531 12593 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:52.487965 16296 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:52.488875 12723 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:52.490334 12988 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.490330 12857 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:52.490607 12662 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca3285abe3654723a6922672391e9423. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.490973 15039 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.491256 15039 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.492890 15472 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.495404 12857 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.495934 12988 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.498620 12450 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index -1 to 4, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:52.512945 16296 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.513432 12593 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.516202 14284 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.518324 12445 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 9 to 10, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.518687 12723 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:52.520459 12855 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:52.521241 15926 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:52.521464 12593 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:52.522010 15926 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:52.525382 13712 raft_consensus.cc:1013] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer a238fc0c6d844cedb8adccb97d01ae5e: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 19:56:52.525775 12661 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.526669 14549 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:52.527439 15926 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:52.529665 15926 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.530849 14549 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.534683 12451 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 10 to 11, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.535610 12855 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.538075 12593 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.548347 15039 raft_consensus.cc:1050] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e to VOTER
I20240501 19:56:52.548910 15039 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:52.553331 16739 ts_tablet_manager.cc:902] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:52.554697 12988 consensus_queue.cc:237] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:52.556617 12723 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 19:56:52.557282 12921 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7ddb6d79d384d39a709b5824f936d53. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.557358 16739 tablet_copy_client.cc:308] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:52.557368 12855 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:52.557498 15508 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.557832 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet cf6948589837482a9f580731e3b861b8 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:57697)
I20240501 19:56:52.557934 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet cf6948589837482a9f580731e3b861b8 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:57697: session id = 2ca06dad25a645bca71b09741798194a-cf6948589837482a9f580731e3b861b8
I20240501 19:56:52.558910 12614 tablet_copy_source_session.cc:205] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.558898 15508 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:52.559826 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6948589837482a9f580731e3b861b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.560441 16196 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.561132 12723 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.563946 12445 catalog_manager.cc:5484] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.567087 12855 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:52.570299 12855 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 19:56:52.570490 14549 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 19:56:52.570765 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:52.571192 12593 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 19:56:52.571619 12661 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.571956 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:52.572229 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:52.576385 15039 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.577386 14549 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.578492 12855 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.580004 12593 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.580924 12450 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 11 to 12, a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.584925 12723 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } }
I20240501 19:56:52.586496 12593 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 19:56:52.586527 12855 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 19:56:52.586541 14549 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 19:56:52.586881 15039 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:52.587047 13712 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:52.587292 13712 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:52.589455 15925 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 19:56:52.590799 16739 tablet_copy_client.cc:783] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.591014 16739 tablet_copy_client.cc:647] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.591459 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8b61e94fc7e541c8aa84957560508a31 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.592702 12445 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 12 to 13, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.593075 12593 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 19:56:52.594686 12857 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 19:56:52.595188 14549 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 19:56:52.595949 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b61e94fc7e541c8aa84957560508a31 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca06dad25a645bca71b09741798194a not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:51964
I20240501 19:56:52.596428 12439 catalog_manager.cc:4830] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): tablet 8b61e94fc7e541c8aa84957560508a31 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
I20240501 19:56:52.598301 16739 tablet_copy_client.cc:523] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:52.599364 12723 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } }
I20240501 19:56:52.599962 16739 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:52.601666 14549 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 19:56:52.601666 12855 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 19:56:52.602000 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:52.602653 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:52.603808 15472 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 19:56:52.605377 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8b61e94fc7e541c8aa84957560508a31 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.605697 12855 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 19:56:52.606958 12450 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 13 to 14, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:52.609136 14549 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 19:56:52.609417 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b61e94fc7e541c8aa84957560508a31 with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:52.609653 16072 tablet_replica.cc:330] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:52.609807 16072 raft_consensus.cc:2229] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.609994 16072 raft_consensus.cc:2258] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.610845 16072 ts_tablet_manager.cc:1875] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.615175 16072 ts_tablet_manager.cc:1888] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.13
I20240501 19:56:52.615254 16072 log.cc:1196] T 8b61e94fc7e541c8aa84957560508a31 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/8b61e94fc7e541c8aa84957560508a31
I20240501 19:56:52.615856 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 8b61e94fc7e541c8aa84957560508a31 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
I20240501 19:56:52.644161 16328 ts_tablet_manager.cc:902] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:52.644639 16328 tablet_copy_client.cc:308] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:52.645303 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 745e39ec23db4f90bb8b1e8b49a50244 from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:44431)
I20240501 19:56:52.645411 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 745e39ec23db4f90bb8b1e8b49a50244 from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:44431: session id = c6d2c7096e404ea9a6f73b123d74c622-745e39ec23db4f90bb8b1e8b49a50244
I20240501 19:56:52.646319 13007 tablet_copy_source_session.cc:205] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.647181 16328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745e39ec23db4f90bb8b1e8b49a50244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.649256 16328 tablet_copy_client.cc:783] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.649399 16328 tablet_copy_client.cc:647] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.654599 12723 consensus_queue.cc:237] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:52.656278 12855 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 19:56:52.656338 12661 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61bc22cdc6ab45529066fe6f4d14d4c7. This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:52.656673 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61bc22cdc6ab45529066fe6f4d14d4c7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:52.656777 13712 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.657804 12593 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.658571 15925 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.659894 15039 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.661234 12855 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.663427 12593 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.663997 12451 catalog_manager.cc:5484] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 4 to 5, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:52.671314 16328 tablet_copy_client.cc:523] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:52.672794 16328 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:52.688575 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.693707 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.702013 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.705199 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.711374 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.719570 14679 raft_consensus.cc:1203] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:52.726497 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.746702 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.763052 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:52.779399 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:52.783392 14549 raft_consensus.cc:1203] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:52.796890 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.798538 16739 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:52.799005 16739 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:52.799289 16739 ts_tablet_manager.cc:1367] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.199s user 0.028s sys 0.001s
I20240501 19:56:52.800308 16739 raft_consensus.cc:348] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:52.800529 16739 raft_consensus.cc:729] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:52.800668 16739 consensus_queue.cc:260] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:52.801118 16739 ts_tablet_manager.cc:1398] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:52.801508 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.801848 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-cf6948589837482a9f580731e3b861b8 received from {username='slave'} at 127.5.139.70:57697
I20240501 19:56:52.801923 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session 2ca06dad25a645bca71b09741798194a-cf6948589837482a9f580731e3b861b8 on tablet cf6948589837482a9f580731e3b861b8 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:52.803645 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.807122 16328 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:52.807389 16328 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:52.807536 16328 ts_tablet_manager.cc:1367] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.135s user 0.029s sys 0.000s
I20240501 19:56:52.807924 16328 raft_consensus.cc:348] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.808032 16328 raft_consensus.cc:729] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:52.808125 16328 consensus_queue.cc:260] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.808369 16328 ts_tablet_manager.cc:1398] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:52.808815 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-745e39ec23db4f90bb8b1e8b49a50244 received from {username='slave'} at 127.5.139.72:44431
I20240501 19:56:52.808881 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-745e39ec23db4f90bb8b1e8b49a50244 on tablet 745e39ec23db4f90bb8b1e8b49a50244 with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:52.810073 15569 raft_consensus.cc:1050] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:52.810475 15569 consensus_queue.cc:237] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:52.812510 12855 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.812968 16123 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.814848 14679 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.816298 15654 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.819509 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.819929 12593 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:52.820804 12723 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.821345 12988 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.821835 15568 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.822958 14679 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.823484 15568 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.825479 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.825711 15568 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.828030 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 955edb7c12fa4b32ac58827f131931e3 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.829082 14679 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.831423 12444 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.831800 12988 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.834407 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 955edb7c12fa4b32ac58827f131931e3 with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:52.834956 16123 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.838913 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.843045 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.843065 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.848829 15553 raft_consensus.cc:1050] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:52.849300 15553 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:52.851155 12592 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.851517 15925 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.852073 14549 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.852424 15925 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.853296 16123 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.855551 14284 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.856736 14680 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.859709 15925 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.859817 12444 catalog_manager.cc:5484] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.862740 12855 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.865420 12593 consensus_queue.cc:237] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:52.867437 14680 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.867841 15569 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.868474 12857 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:52.869477 15569 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:52.870352 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.870908 15569 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.872370 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 544caa38514643d5b81cd6ed85e0bae7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:52.873219 12857 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.877266 12444 catalog_manager.cc:5484] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.880198 16328 ts_tablet_manager.cc:902] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:52.880681 16328 tablet_copy_client.cc:308] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:52.881107 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 04ccc8addc55441b9ab9d8265b90b03b from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:37741)
I20240501 19:56:52.881199 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 04ccc8addc55441b9ab9d8265b90b03b from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:37741: session id = c6d2c7096e404ea9a6f73b123d74c622-04ccc8addc55441b9ab9d8265b90b03b
I20240501 19:56:52.881275 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet 544caa38514643d5b81cd6ed85e0bae7 with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:52.882143 12877 tablet_copy_source_session.cc:205] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.882925 16328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ccc8addc55441b9ab9d8265b90b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.884896 16328 tablet_copy_client.cc:783] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.885021 16328 tablet_copy_client.cc:647] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.888265 14680 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:52.891973 16886 tablet_replica.cc:330] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:52.892383 16886 raft_consensus.cc:2229] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.892525 16328 tablet_copy_client.cc:523] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:52.893718 16328 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:52.897282 16886 pending_rounds.cc:62] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Trying to abort 1 pending ops.
I20240501 19:56:52.897437 16886 pending_rounds.cc:68] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022974618870669312 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "544caa38514643d5b81cd6ed85e0bae7" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } } }
I20240501 19:56:52.897708 16886 raft_consensus.cc:2894] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 19:56:52.897830 16886 raft_consensus.cc:2866] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:52.897928 16886 raft_consensus.cc:2258] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.902915 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:52.907987 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.913105 16890 ts_tablet_manager.cc:902] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:52.913713 16890 tablet_copy_client.cc:308] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:52.914261 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 0fb481d394034a8f85af613beb5c9ec8 from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:59875)
I20240501 19:56:52.914355 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 0fb481d394034a8f85af613beb5c9ec8 from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:59875: session id = 9069ec6fcdb248479037f2d37e0e851d-0fb481d394034a8f85af613beb5c9ec8
I20240501 19:56:52.914459 15926 raft_consensus.cc:1050] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:52.914804 15926 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:52.915212 12877 tablet_copy_source_session.cc:205] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.916108 12988 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.916134 14416 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.916641 14548 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:52.916810 16890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0fb481d394034a8f85af613beb5c9ec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.916955 15472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.917620 15472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.918920 15472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:52.920122 16890 tablet_copy_client.cc:783] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.920313 16890 tablet_copy_client.cc:647] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.925341 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:52.933727 15508 raft_consensus.cc:1050] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:52.934152 15508 consensus_queue.cc:237] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:52.935462 14680 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.935709 12857 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.935889 15488 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.936002 12592 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:52.936023 16887 ts_tablet_manager.cc:902] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:52.936551 16295 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.936682 16887 tablet_copy_client.cc:308] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:52.936841 16295 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:52.937458 12745 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet ca3285abe3654723a6922672391e9423 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:41689)
I20240501 19:56:52.937546 12745 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet ca3285abe3654723a6922672391e9423 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:41689: session id = 1d49e80be3d04fb99b711c62d61bac09-ca3285abe3654723a6922672391e9423
I20240501 19:56:52.938414 12745 tablet_copy_source_session.cc:205] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.938506 15553 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.939117 16328 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:52.939396 16328 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:52.939577 16328 ts_tablet_manager.cc:1367] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.046s user 0.026s sys 0.000s
I20240501 19:56:52.940083 16328 raft_consensus.cc:348] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.940224 16328 raft_consensus.cc:729] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:52.940337 16328 consensus_queue.cc:260] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.940655 16328 ts_tablet_manager.cc:1398] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:52.940663 16890 tablet_copy_client.cc:523] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:52.941082 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-04ccc8addc55441b9ab9d8265b90b03b received from {username='slave'} at 127.5.139.72:37741
I20240501 19:56:52.941156 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-04ccc8addc55441b9ab9d8265b90b03b on tablet 04ccc8addc55441b9ab9d8265b90b03b with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:52.942122 16890 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:52.944249 12453 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:52.945483 14548 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.945631 14416 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.949278 16887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3285abe3654723a6922672391e9423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.949940 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.951754 16887 tablet_copy_client.cc:783] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.951903 16887 tablet_copy_client.cc:647] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.961814 16875 tablet_replica.cc:330] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:52.962000 16875 raft_consensus.cc:2229] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:52.962209 16875 raft_consensus.cc:2258] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:52.964854 16875 ts_tablet_manager.cc:1875] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:52.967283 16875 ts_tablet_manager.cc:1888] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:52.967520 16875 log.cc:1196] T 955edb7c12fa4b32ac58827f131931e3 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/955edb7c12fa4b32ac58827f131931e3
I20240501 19:56:52.968081 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 955edb7c12fa4b32ac58827f131931e3 (table rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]) successfully deleted
I20240501 19:56:52.975306 15553 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.975651 14284 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.977990 12593 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.978540 12444 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:52.979882 16889 ts_tablet_manager.cc:902] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:52.983268 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:52.986327 16881 ts_tablet_manager.cc:902] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:52.986842 16881 tablet_copy_client.cc:308] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:52.987337 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:52.987682 16328 ts_tablet_manager.cc:902] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:52.989344 16898 ts_tablet_manager.cc:902] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:52.989797 16898 tablet_copy_client.cc:308] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:52.990177 12745 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 12a50be843fb49ee8a66c9a17d271d61 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:41689)
I20240501 19:56:52.990051 12988 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.990270 12745 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 12a50be843fb49ee8a66c9a17d271d61 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:41689: session id = 1d49e80be3d04fb99b711c62d61bac09-12a50be843fb49ee8a66c9a17d271d61
I20240501 19:56:52.990748 16328 tablet_copy_client.cc:308] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:52.991187 12745 tablet_copy_source_session.cc:205] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.992835 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet d75bd8f836b947248ee124f456c7d721 from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:37741)
I20240501 19:56:52.992954 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet d75bd8f836b947248ee124f456c7d721 from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:37741: session id = c6d2c7096e404ea9a6f73b123d74c622-d75bd8f836b947248ee124f456c7d721
I20240501 19:56:52.993865 12877 tablet_copy_source_session.cc:205] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:52.994292 16862 ts_tablet_manager.cc:902] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:52.995007 16328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d75bd8f836b947248ee124f456c7d721. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:52.995522 14549 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:52.997803 16328 tablet_copy_client.cc:783] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:52.997946 16328 tablet_copy_client.cc:647] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:52.998153 16862 tablet_copy_client.cc:308] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
W20240501 19:56:52.998653 16876 consensus_queue.cc:1159] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" status { last_received { term: 1 index: 5 } last_committed_idx: 4 last_received_current_leader { term: 1 index: 5 } }
I20240501 19:56:52.999640 16886 ts_tablet_manager.cc:1875] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:53.001282 16887 tablet_copy_client.cc:523] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.001683 16886 ts_tablet_manager.cc:1888] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:53.001742 16886 log.cc:1196] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/544caa38514643d5b81cd6ed85e0bae7
I20240501 19:56:53.002473 16887 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:53.003659 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 557994658e2c4c8bbc1eeacdab958ddd from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:59875)
I20240501 19:56:53.003751 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 557994658e2c4c8bbc1eeacdab958ddd from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:59875: session id = 9069ec6fcdb248479037f2d37e0e851d-557994658e2c4c8bbc1eeacdab958ddd
I20240501 19:56:53.003686 15488 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.004626 12877 tablet_copy_source_session.cc:205] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.005602 14680 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.005653 12857 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.006235 16898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a50be843fb49ee8a66c9a17d271d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.006837 16862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557994658e2c4c8bbc1eeacdab958ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.008141 12444 catalog_manager.cc:5484] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.009912 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet 544caa38514643d5b81cd6ed85e0bae7 (table rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]) successfully deleted
I20240501 19:56:53.011740 12592 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.017848 16889 tablet_copy_client.cc:308] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:53.018333 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet 7d7ea564ea504163a1e17561be9a5130 from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:60521)
I20240501 19:56:53.018427 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet 7d7ea564ea504163a1e17561be9a5130 from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:60521: session id = 9069ec6fcdb248479037f2d37e0e851d-7d7ea564ea504163a1e17561be9a5130
I20240501 19:56:53.019263 12614 tablet_copy_source_session.cc:205] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.020121 16889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7ea564ea504163a1e17561be9a5130. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.024924 16328 tablet_copy_client.cc:523] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.026016 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet d940b16d07d54945abe4a10a11dd6a2a from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:49353)
I20240501 19:56:53.026124 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet d940b16d07d54945abe4a10a11dd6a2a from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:49353: session id = 1d49e80be3d04fb99b711c62d61bac09-d940b16d07d54945abe4a10a11dd6a2a
I20240501 19:56:53.026306 16328 tablet_bootstrap.cc:492] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:53.026959 12614 tablet_copy_source_session.cc:205] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.028256 16881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940b16d07d54945abe4a10a11dd6a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.028789 16862 tablet_copy_client.cc:783] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.028940 16862 tablet_copy_client.cc:647] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.030547 16889 tablet_copy_client.cc:783] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.030762 16889 tablet_copy_client.cc:647] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.032212 14680 raft_consensus.cc:1203] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:53.033370 16889 tablet_copy_client.cc:523] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.035020 16889 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:53.041221 16898 tablet_copy_client.cc:783] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.041391 16898 tablet_copy_client.cc:647] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.044661 16881 tablet_copy_client.cc:783] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.044838 16881 tablet_copy_client.cc:647] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.046276 16881 tablet_copy_client.cc:523] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.047549 16881 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:53.047580 16739 ts_tablet_manager.cc:902] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:53.048018 16739 tablet_copy_client.cc:308] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:53.048454 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 557994658e2c4c8bbc1eeacdab958ddd from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:54927)
I20240501 19:56:53.048556 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 557994658e2c4c8bbc1eeacdab958ddd from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:54927: session id = 2ca06dad25a645bca71b09741798194a-557994658e2c4c8bbc1eeacdab958ddd
I20240501 19:56:53.049449 12877 tablet_copy_source_session.cc:205] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.050518 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557994658e2c4c8bbc1eeacdab958ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.052551 16739 tablet_copy_client.cc:783] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.052690 16739 tablet_copy_client.cc:647] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.059199 16947 ts_tablet_manager.cc:902] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:53.059877 16947 tablet_copy_client.cc:308] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:53.062126 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:41689)
I20240501 19:56:53.062216 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:41689: session id = 1d49e80be3d04fb99b711c62d61bac09-800d5f43cd1b4fb6bd8bbcac262aeb80
I20240501 19:56:53.063067 12746 tablet_copy_source_session.cc:205] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.064505 16898 tablet_copy_client.cc:523] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.065899 16898 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:53.070420 16890 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.071795 16890 tablet_bootstrap.cc:492] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:53.072043 16890 ts_tablet_manager.cc:1367] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.130s user 0.028s sys 0.000s
I20240501 19:56:53.072535 16890 raft_consensus.cc:348] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.072759 16890 raft_consensus.cc:729] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:53.072906 16890 consensus_queue.cc:260] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.073323 16890 ts_tablet_manager.cc:1398] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.073899 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-0fb481d394034a8f85af613beb5c9ec8 received from {username='slave'} at 127.5.139.69:59875
I20240501 19:56:53.073971 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-0fb481d394034a8f85af613beb5c9ec8 on tablet 0fb481d394034a8f85af613beb5c9ec8 with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:53.074803 16958 ts_tablet_manager.cc:902] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:53.075232 16958 tablet_copy_client.cc:308] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:53.075636 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 61bc22cdc6ab45529066fe6f4d14d4c7 from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:47601)
I20240501 19:56:53.075733 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 61bc22cdc6ab45529066fe6f4d14d4c7 from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:47601: session id = 9069ec6fcdb248479037f2d37e0e851d-61bc22cdc6ab45529066fe6f4d14d4c7
I20240501 19:56:53.076331 16862 tablet_copy_client.cc:523] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.076609 12746 tablet_copy_source_session.cc:205] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.077972 16862 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:53.081331 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61bc22cdc6ab45529066fe6f4d14d4c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.081904 16978 ts_tablet_manager.cc:902] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:53.082469 16978 tablet_copy_client.cc:308] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:53.082975 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 706878734216445f8422e422228d4230 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:50147)
I20240501 19:56:53.083096 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 706878734216445f8422e422228d4230 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:50147: session id = 2ca06dad25a645bca71b09741798194a-706878734216445f8422e422228d4230
I20240501 19:56:53.083909 16958 tablet_copy_client.cc:783] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.084033 16958 tablet_copy_client.cc:647] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.084044 13007 tablet_copy_source_session.cc:205] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.084877 16978 ts_tablet_manager.cc:1875] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:56:53.087064 16978 ts_tablet_manager.cc:1888] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 19:56:53.087289 16978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706878734216445f8422e422228d4230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.087530 16947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.088589 16978 tablet_copy_client.cc:783] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.088719 16978 tablet_copy_client.cc:647] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.089736 16978 tablet_copy_client.cc:523] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.090885 16947 tablet_copy_client.cc:783] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.091063 16947 tablet_copy_client.cc:647] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.091394 16978 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:53.100061 16898 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.100402 16898 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:53.100598 16898 ts_tablet_manager.cc:1367] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.035s user 0.028s sys 0.000s
I20240501 19:56:53.101058 16898 raft_consensus.cc:348] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:53.101255 16898 raft_consensus.cc:729] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:53.101392 16898 consensus_queue.cc:260] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:53.101784 16898 ts_tablet_manager.cc:1398] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.102638 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-12a50be843fb49ee8a66c9a17d271d61 received from {username='slave'} at 127.5.139.71:41689
I20240501 19:56:53.102715 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-12a50be843fb49ee8a66c9a17d271d61 on tablet 12a50be843fb49ee8a66c9a17d271d61 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:53.104990 16328 tablet_bootstrap.cc:492] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.105299 16328 tablet_bootstrap.cc:492] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:53.105468 16328 ts_tablet_manager.cc:1367] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.079s user 0.017s sys 0.008s
I20240501 19:56:53.105544 16958 tablet_copy_client.cc:523] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.105921 16328 raft_consensus.cc:348] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:53.106041 16328 raft_consensus.cc:729] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:53.106139 16328 consensus_queue.cc:260] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:53.106437 16328 ts_tablet_manager.cc:1398] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.107110 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-d75bd8f836b947248ee124f456c7d721 received from {username='slave'} at 127.5.139.72:37741
I20240501 19:56:53.107198 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-d75bd8f836b947248ee124f456c7d721 on tablet d75bd8f836b947248ee124f456c7d721 with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:53.107774 16958 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:53.116644 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.117398 16889 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.118018 16889 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:53.118233 16889 ts_tablet_manager.cc:1367] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.083s user 0.030s sys 0.000s
I20240501 19:56:53.118840 16889 raft_consensus.cc:348] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.119022 16889 raft_consensus.cc:729] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:53.119165 16889 consensus_queue.cc:260] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.119479 16889 ts_tablet_manager.cc:1398] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.119989 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-7d7ea564ea504163a1e17561be9a5130 received from {username='slave'} at 127.5.139.69:60521
I20240501 19:56:53.120064 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-7d7ea564ea504163a1e17561be9a5130 on tablet 7d7ea564ea504163a1e17561be9a5130 with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:53.132683 14416 raft_consensus.cc:1203] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:53.135084 16881 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.135465 16881 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:53.135686 16881 ts_tablet_manager.cc:1367] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.088s user 0.028s sys 0.000s
I20240501 19:56:53.136143 16881 raft_consensus.cc:348] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:53.136369 16881 raft_consensus.cc:729] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:53.136518 16881 consensus_queue.cc:260] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:53.136866 16881 ts_tablet_manager.cc:1398] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.137517 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-d940b16d07d54945abe4a10a11dd6a2a received from {username='slave'} at 127.5.139.71:49353
I20240501 19:56:53.137605 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-d940b16d07d54945abe4a10a11dd6a2a on tablet d940b16d07d54945abe4a10a11dd6a2a with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:53.142223 16978 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.142586 16978 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:53.142798 16978 ts_tablet_manager.cc:1367] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.051s user 0.024s sys 0.005s
I20240501 19:56:53.143298 16978 raft_consensus.cc:348] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.143464 16978 raft_consensus.cc:729] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:53.143571 16978 consensus_queue.cc:260] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.143890 16978 ts_tablet_manager.cc:1398] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:53.144341 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-706878734216445f8422e422228d4230 received from {username='slave'} at 127.5.139.70:50147
I20240501 19:56:53.144419 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 2ca06dad25a645bca71b09741798194a-706878734216445f8422e422228d4230 on tablet 706878734216445f8422e422228d4230 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:53.150279 16958 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.150568 16958 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:53.150750 16958 ts_tablet_manager.cc:1367] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.043s user 0.025s sys 0.000s
I20240501 19:56:53.151203 16958 raft_consensus.cc:348] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.151335 16958 raft_consensus.cc:729] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:53.151461 16958 consensus_queue.cc:260] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.154803 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.155340 16958 ts_tablet_manager.cc:1398] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240501 19:56:53.156632 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.156821 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-61bc22cdc6ab45529066fe6f4d14d4c7 received from {username='slave'} at 127.5.139.69:47601
I20240501 19:56:53.156905 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-61bc22cdc6ab45529066fe6f4d14d4c7 on tablet 61bc22cdc6ab45529066fe6f4d14d4c7 with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:53.166113 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:53.166394 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.168944 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.172278 16958 ts_tablet_manager.cc:902] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:53.175093 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.179854 16958 tablet_copy_client.cc:308] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:53.180303 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet c7ddb6d79d384d39a709b5824f936d53 from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:43093)
I20240501 19:56:53.180406 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet c7ddb6d79d384d39a709b5824f936d53 from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:43093: session id = 9069ec6fcdb248479037f2d37e0e851d-c7ddb6d79d384d39a709b5824f936d53
I20240501 19:56:53.181342 13007 tablet_copy_source_session.cc:205] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.183763 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7ddb6d79d384d39a709b5824f936d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.185367 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.187805 16958 tablet_copy_client.cc:783] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.188028 16958 tablet_copy_client.cc:647] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.195426 16947 tablet_copy_client.cc:523] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.197256 16947 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:53.214453 16739 tablet_copy_client.cc:523] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.216007 16739 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:53.226943 16653 raft_consensus.cc:1050] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:53.227449 16653 consensus_queue.cc:237] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:53.229279 14680 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.229732 16064 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.231470 12986 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.232028 14549 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.232318 16655 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.233134 16064 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.234791 12723 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.235255 16064 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.235904 16064 raft_consensus.cc:1013] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: attempt to promote peer c6d2c7096e404ea9a6f73b123d74c622: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 19:56:53.246953 16947 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.247314 16947 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:53.247511 16947 ts_tablet_manager.cc:1367] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.050s user 0.027s sys 0.000s
I20240501 19:56:53.247956 16947 raft_consensus.cc:348] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:53.248092 16947 raft_consensus.cc:729] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:53.248194 16947 consensus_queue.cc:260] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:53.248461 16947 ts_tablet_manager.cc:1398] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.249003 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-800d5f43cd1b4fb6bd8bbcac262aeb80 received from {username='slave'} at 127.5.139.71:41689
I20240501 19:56:53.249148 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-800d5f43cd1b4fb6bd8bbcac262aeb80 on tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:53.249958 16739 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.250245 16739 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:53.250452 16739 ts_tablet_manager.cc:1367] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.035s user 0.023s sys 0.004s
I20240501 19:56:53.250943 16739 raft_consensus.cc:348] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.251081 16739 raft_consensus.cc:729] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:53.251196 16739 consensus_queue.cc:260] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.252156 16739 ts_tablet_manager.cc:1398] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:53.252677 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-557994658e2c4c8bbc1eeacdab958ddd received from {username='slave'} at 127.5.139.70:54927
I20240501 19:56:53.252764 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session 2ca06dad25a645bca71b09741798194a-557994658e2c4c8bbc1eeacdab958ddd on tablet 557994658e2c4c8bbc1eeacdab958ddd with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:53.253882 16958 tablet_copy_client.cc:523] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.255404 16958 tablet_bootstrap.cc:492] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:53.255232 16653 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.258688 14680 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.259104 12444 catalog_manager.cc:5484] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:53.260758 12986 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.260756 14549 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.262881 16064 raft_consensus.cc:1050] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:53.263355 16064 consensus_queue.cc:237] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:53.265473 14549 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:53.265491 12986 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:53.265934 16655 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:53.266445 16655 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:53.266899 14680 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:53.267313 16655 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:53.268904 16739 ts_tablet_manager.cc:902] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:53.269383 16739 tablet_copy_client.cc:308] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:53.269912 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 61bc22cdc6ab45529066fe6f4d14d4c7 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:36001)
I20240501 19:56:53.270020 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 61bc22cdc6ab45529066fe6f4d14d4c7 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:36001: session id = 2ca06dad25a645bca71b09741798194a-61bc22cdc6ab45529066fe6f4d14d4c7
I20240501 19:56:53.270897 12746 tablet_copy_source_session.cc:205] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.270938 12725 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:53.271741 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61bc22cdc6ab45529066fe6f4d14d4c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.274009 16739 tablet_copy_client.cc:783] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.274156 16739 tablet_copy_client.cc:647] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.274271 17014 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.275075 12725 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.275764 16655 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.277783 14680 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.278178 12986 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.279886 16739 tablet_copy_client.cc:523] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.280443 12450 catalog_manager.cc:5484] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 6 to 7, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.280476 14549 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.281282 16739 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:53.285952 12857 consensus_queue.cc:237] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:53.287201 14549 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.287827 16655 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.288810 12986 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.290205 14680 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.290589 17039 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.290746 17014 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.292307 16655 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.293687 14680 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.294360 14549 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.294868 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bf64d9f3ce94db28b032cdcd795a255 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:53.296548 12453 catalog_manager.cc:5484] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 7 to 8, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.298267 12986 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.299769 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf64d9f3ce94db28b032cdcd795a255 with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:53.299963 16886 tablet_replica.cc:330] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:53.300097 16886 raft_consensus.cc:2229] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:53.300243 16886 pending_rounds.cc:62] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: Trying to abort 1 pending ops.
I20240501 19:56:53.300271 16886 pending_rounds.cc:68] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7022974620725526528 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "0bf64d9f3ce94db28b032cdcd795a255" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } } }
I20240501 19:56:53.300402 16886 raft_consensus.cc:2894] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240501 19:56:53.300460 16886 raft_consensus.cc:2866] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:53.300501 16886 raft_consensus.cc:2258] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:53.303086 16886 ts_tablet_manager.cc:1875] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:53.303848 12857 consensus_queue.cc:237] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:53.306099 16886 ts_tablet_manager.cc:1888] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:53.306171 16862 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.306180 16886 log.cc:1196] T 0bf64d9f3ce94db28b032cdcd795a255 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/0bf64d9f3ce94db28b032cdcd795a255
I20240501 19:56:53.306417 16862 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:53.306892 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet 0bf64d9f3ce94db28b032cdcd795a255 (table rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]) successfully deleted
I20240501 19:56:53.307536 14680 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.307895 16655 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.309612 14549 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.310088 16655 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.311352 17039 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.312791 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bf64d9f3ce94db28b032cdcd795a255 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:53.314249 12453 catalog_manager.cc:5484] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 8 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:53.315604 14680 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.306615 16862 ts_tablet_manager.cc:1367] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.229s user 0.023s sys 0.004s
I20240501 19:56:53.317603 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bf64d9f3ce94db28b032cdcd795a255 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:53.317589 16862 raft_consensus.cc:348] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.317868 16862 raft_consensus.cc:729] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:53.318032 16862 consensus_queue.cc:260] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.320801 14549 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.319620 14284 raft_consensus.cc:1203] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 19:56:53.321604 16862 ts_tablet_manager.cc:1398] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:53.322137 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-557994658e2c4c8bbc1eeacdab958ddd received from {username='slave'} at 127.5.139.69:59875
I20240501 19:56:53.322208 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-557994658e2c4c8bbc1eeacdab958ddd on tablet 557994658e2c4c8bbc1eeacdab958ddd with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:53.332412 12725 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:53.334619 14416 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 19:56:53.334699 12661 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4b15761177f4cffb7aa5e4d16e42ac7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.334921 14549 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.335052 15926 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.335528 15926 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.336962 12986 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.337649 15926 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.339272 15039 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.339695 12986 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.341465 14416 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.341697 14284 raft_consensus.cc:1203] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:53.343127 12445 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 8 to 9, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.343300 16958 tablet_bootstrap.cc:492] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.343561 16958 tablet_bootstrap.cc:492] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:53.343737 16958 ts_tablet_manager.cc:1367] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.088s user 0.026s sys 0.000s
I20240501 19:56:53.344245 16958 raft_consensus.cc:348] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.344359 16958 raft_consensus.cc:729] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:53.344465 16958 consensus_queue.cc:260] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:53.344720 16958 ts_tablet_manager.cc:1398] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:53.345016 16887 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.345124 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-c7ddb6d79d384d39a709b5824f936d53 received from {username='slave'} at 127.5.139.69:43093
I20240501 19:56:53.345198 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-c7ddb6d79d384d39a709b5824f936d53 on tablet c7ddb6d79d384d39a709b5824f936d53 with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:53.346403 14549 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.350312 16887 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:53.351323 16887 ts_tablet_manager.cc:1367] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.349s user 0.028s sys 0.001s
I20240501 19:56:53.352164 16887 raft_consensus.cc:348] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:53.352383 12725 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:53.352360 16887 raft_consensus.cc:729] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
W20240501 19:56:53.353799 12661 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4b15761177f4cffb7aa5e4d16e42ac7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.354069 14549 raft_consensus.cc:1203] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:53.354259 14548 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:53.354859 15553 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.354979 16887 consensus_queue.cc:260] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:53.355351 16739 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.355432 16887 ts_tablet_manager.cc:1398] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 19:56:53.355612 16739 tablet_bootstrap.cc:492] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:53.355772 16739 ts_tablet_manager.cc:1367] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.075s user 0.028s sys 0.001s
I20240501 19:56:53.355937 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-ca3285abe3654723a6922672391e9423 received from {username='slave'} at 127.5.139.71:41689
I20240501 19:56:53.356009 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-ca3285abe3654723a6922672391e9423 on tablet ca3285abe3654723a6922672391e9423 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:53.356204 16739 raft_consensus.cc:348] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.356319 16739 raft_consensus.cc:729] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:53.356415 16739 consensus_queue.cc:260] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.356524 14416 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:53.356935 15926 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.357383 16739 ts_tablet_manager.cc:1398] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 19:56:53.358146 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-61bc22cdc6ab45529066fe6f4d14d4c7 received from {username='slave'} at 127.5.139.70:36001
I20240501 19:56:53.358217 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2ca06dad25a645bca71b09741798194a-61bc22cdc6ab45529066fe6f4d14d4c7 on tablet 61bc22cdc6ab45529066fe6f4d14d4c7 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:53.358631 13712 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.360555 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4b15761177f4cffb7aa5e4d16e42ac7 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:53.360652 14549 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.360810 14416 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.363798 12451 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 9 to 10, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:53.367074 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b15761177f4cffb7aa5e4d16e42ac7 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:53.372215 14549 raft_consensus.cc:1203] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:53.375452 17045 tablet_replica.cc:330] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:53.376489 17045 raft_consensus.cc:2229] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:53.377135 17045 raft_consensus.cc:2258] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:53.377933 17045 ts_tablet_manager.cc:1875] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:53.380240 17045 ts_tablet_manager.cc:1888] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:53.380370 17045 log.cc:1196] T 0bf64d9f3ce94db28b032cdcd795a255 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/0bf64d9f3ce94db28b032cdcd795a255
I20240501 19:56:53.380952 17045 tablet_replica.cc:330] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:53.381076 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 0bf64d9f3ce94db28b032cdcd795a255 (table rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]) successfully deleted
I20240501 19:56:53.381235 17045 raft_consensus.cc:2229] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:53.381507 17045 raft_consensus.cc:2258] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:53.382493 17045 ts_tablet_manager.cc:1875] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:53.384474 17045 ts_tablet_manager.cc:1888] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 19:56:53.386794 17045 log.cc:1196] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/e4b15761177f4cffb7aa5e4d16e42ac7
I20240501 19:56:53.387346 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet e4b15761177f4cffb7aa5e4d16e42ac7 (table rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]) successfully deleted
I20240501 19:56:53.389250 12725 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:53.390942 14284 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.391256 12986 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.391335 12857 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.391598 15553 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.393216 15553 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.393545 15553 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.396910 15926 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.397230 12723 consensus_queue.cc:237] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.399246 12988 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:53.399324 14284 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.399652 15926 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 19:56:53.401556 12855 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.401685 12592 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 19:56:53.402354 15039 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 19:56:53.403995 12661 consensus_peers.cc:485] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae2885f7933f462c96db9a99a4eb2090. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.404464 12450 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 7 to 8. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.405172 14549 raft_consensus.cc:1203] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:53.406989 13712 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.410360 12988 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.412114 12723 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.413749 12855 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.413774 14284 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.414515 15925 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.414623 12452 catalog_manager.cc:5484] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.414806 15925 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.415843 12986 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.417066 15472 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.417284 14549 raft_consensus.cc:1203] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:53.419075 12986 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.421715 12592 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
W20240501 19:56:53.422971 12661 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e84a3865ce64eaebad4df46ec01842d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.423998 15925 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.424345 12855 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.426173 14284 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.427109 12444 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.427325 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3e84a3865ce64eaebad4df46ec01842d with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 19:56:53.427274 12986 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.435933 12723 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.437647 12855 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 19:56:53.437779 12661 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e84a3865ce64eaebad4df46ec01842d. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.437857 14284 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:53.438053 15926 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.438292 15472 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.439554 13712 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.440518 12855 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.440698 14284 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.443228 12451 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 9 to 10, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.446202 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e84a3865ce64eaebad4df46ec01842d with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:53.447747 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e84a3865ce64eaebad4df46ec01842d with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:53.447966 17045 tablet_replica.cc:330] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:53.448105 17045 raft_consensus.cc:2229] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:53.448292 17045 raft_consensus.cc:2258] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:53.449213 17045 ts_tablet_manager.cc:1875] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:53.450644 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.451956 17045 ts_tablet_manager.cc:1888] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 19:56:53.463070 14284 raft_consensus.cc:1203] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:53.466423 17045 log.cc:1196] T 3e84a3865ce64eaebad4df46ec01842d P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/3e84a3865ce64eaebad4df46ec01842d
I20240501 19:56:53.467064 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 3e84a3865ce64eaebad4df46ec01842d (table rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]) successfully deleted
I20240501 19:56:53.470597 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.472954 14680 raft_consensus.cc:1203] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:53.478377 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:53.483644 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.487036 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.487891 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.489391 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.502369 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.505187 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.510591 12986 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.512912 12855 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:53.513870 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.516366 14549 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 19:56:53.516644 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.516975 16296 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.520064 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.520309 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.520716 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.526185 16296 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.529903 14549 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.532605 12445 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 9 to 10, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.542101 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.547452 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.557809 14284 raft_consensus.cc:1203] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:53.566941 15568 raft_consensus.cc:1050] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:53.567345 15568 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:53.568910 14415 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.569345 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.569577 15654 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.569749 12857 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.570094 15654 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.571200 14284 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.571600 15654 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.574546 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:53.574996 12986 consensus_queue.cc:237] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:53.577998 14285 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:53.578302 12856 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:53.579862 15508 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:53.580143 15508 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:53.582124 16296 raft_consensus.cc:1050] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:53.582475 16296 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:53.583981 12725 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.584038 14679 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.584349 16196 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.584653 16196 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.585321 14286 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.585737 16196 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.586803 12723 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:53.587181 15508 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 19:56:53.587781 12923 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7ddb6d79d384d39a709b5824f936d53. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.589500 15508 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.591434 12723 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.593528 12445 catalog_manager.cc:5484] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 4 to 5, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:53.598906 14680 raft_consensus.cc:1203] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:53.602243 14416 raft_consensus.cc:1203] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:53.615859 15654 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:53.616731 16296 raft_consensus.cc:1050] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:53.617301 16296 consensus_queue.cc:237] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:53.619598 12723 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.619730 14285 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.620204 16196 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.620476 16196 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:53.620563 12444 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 7 to 8, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:53.621327 14285 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.622004 12857 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.623347 15508 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:53.623495 14284 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:53.624063 12857 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.626860 16196 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.627287 14415 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:53.629770 12723 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.629921 14285 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
W20240501 19:56:53.623354 12923 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c7ddb6d79d384d39a709b5824f936d53. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.632138 12449 catalog_manager.cc:5484] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:53.633720 12855 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:53.633824 12857 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.633788 12856 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:53.639123 16296 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.641463 12725 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.642205 14679 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.644147 14286 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.644593 12444 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 7 to 8, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.648739 12986 consensus_queue.cc:237] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:53.650826 12725 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:53.651105 12986 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:53.651217 16196 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:53.651815 12856 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:53.652294 14679 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.652690 14285 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.652928 17157 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:53.653544 14286 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:53.653128 17158 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.653903 16196 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.653991 15508 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:53.655858 14415 raft_consensus.cc:1203] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 19:56:53.657637 16196 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.659638 16196 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.660077 14679 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.661193 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c7ddb6d79d384d39a709b5824f936d53 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:53.661356 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 745e39ec23db4f90bb8b1e8b49a50244 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:53.662420 12445 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.663164 12856 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.663487 12725 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.665962 12446 catalog_manager.cc:5484] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.667369 14285 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.669667 14660 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ddb6d79d384d39a709b5824f936d53 with delete_type TABLET_DATA_TOMBSTONED (TS c6d2c7096e404ea9a6f73b123d74c622 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:52566
I20240501 19:56:53.670248 14286 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:53.671599 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet 745e39ec23db4f90bb8b1e8b49a50244 with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:53.671792 16886 tablet_replica.cc:330] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:53.671921 16886 raft_consensus.cc:2229] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:53.672092 16886 raft_consensus.cc:2258] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:53.672885 16886 ts_tablet_manager.cc:1875] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:53.675074 16886 ts_tablet_manager.cc:1888] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:53.675168 16886 log.cc:1196] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/745e39ec23db4f90bb8b1e8b49a50244
I20240501 19:56:53.675630 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet 745e39ec23db4f90bb8b1e8b49a50244 (table rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]) successfully deleted
W20240501 19:56:53.676237 12439 catalog_manager.cc:4797] TS c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): delete failed for tablet c7ddb6d79d384d39a709b5824f936d53 because the tablet was not found. No further retry: Not found: Tablet not found: c7ddb6d79d384d39a709b5824f936d53
I20240501 19:56:53.690374 12592 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:53.692368 12986 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.692384 14679 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.692773 15569 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 19:56:53.693253 12531 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 955edb7c12fa4b32ac58827f131931e3. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.694253 15569 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.695755 15569 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:53.699821 12986 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:53.702404 12446 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 8 to 9, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:53.702492 14679 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 19:56:53.748144 12856 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.750383 14679 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.751206 14286 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.751258 12723 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.751696 16655 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.751998 16655 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.753742 17014 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 19:56:53.754470 12792 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04ccc8addc55441b9ab9d8265b90b03b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.755373 17014 raft_consensus.cc:1050] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:53.755724 17014 consensus_queue.cc:237] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:53.757691 12593 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:53.757917 14677 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:53.758380 14285 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:53.759253 17039 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.760475 17015 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.760780 17015 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.762825 14415 raft_consensus.cc:1203] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 19:56:53.768595 17233 ts_tablet_manager.cc:902] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:53.769099 17233 tablet_copy_client.cc:308] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:53.769515 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet e4b15761177f4cffb7aa5e4d16e42ac7 from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:60977)
I20240501 19:56:53.769621 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet e4b15761177f4cffb7aa5e4d16e42ac7 from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:60977: session id = c6d2c7096e404ea9a6f73b123d74c622-e4b15761177f4cffb7aa5e4d16e42ac7
I20240501 19:56:53.770502 12746 tablet_copy_source_session.cc:205] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.772553 17233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b15761177f4cffb7aa5e4d16e42ac7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.772651 16655 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.773896 16064 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.776067 12593 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.776105 14285 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.778049 12723 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.778203 14680 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.778815 12446 catalog_manager.cc:5484] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 9 to 10, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.780983 12445 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 7 to 8, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.783260 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.786550 12856 consensus_queue.cc:237] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:53.788971 14680 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:53.789638 14286 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.792515 17015 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:53.795688 14286 raft_consensus.cc:1261] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:53.796474 17039 consensus_queue.cc:1035] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:53.797458 17039 raft_consensus.cc:1050] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:53.797847 17039 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.799667 17233 tablet_copy_client.cc:783] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.799826 17233 tablet_copy_client.cc:647] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.800981 14677 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.801175 14679 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.802845 14285 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.803227 12723 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 19:56:53.804265 12792 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04ccc8addc55441b9ab9d8265b90b03b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.804566 17015 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.805169 17015 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.806459 17015 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.807832 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.811427 17015 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.814550 14679 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.814620 16064 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.817077 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0fb481d394034a8f85af613beb5c9ec8 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:53.817440 14680 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.817559 14285 raft_consensus.cc:2932] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.819164 12445 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 8 to 9, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:53.819809 14286 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.820657 12453 catalog_manager.cc:5484] T 0fb481d394034a8f85af613beb5c9ec8 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 10 to 11, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.822657 12723 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.824573 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet 0fb481d394034a8f85af613beb5c9ec8 with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:53.826308 17233 tablet_copy_client.cc:523] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.827725 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.831029 16739 ts_tablet_manager.cc:902] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:53.831503 16739 tablet_copy_client.cc:308] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:53.832229 17140 raft_consensus.cc:1050] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:53.832652 17140 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:53.834038 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet ae2885f7933f462c96db9a99a4eb2090 from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:36001)
I20240501 19:56:53.834064 14415 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.834154 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet ae2885f7933f462c96db9a99a4eb2090 from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:36001: session id = 2ca06dad25a645bca71b09741798194a-ae2885f7933f462c96db9a99a4eb2090
I20240501 19:56:53.835018 12746 tablet_copy_source_session.cc:205] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.835654 15654 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.836933 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2885f7933f462c96db9a99a4eb2090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.837332 14285 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.837723 15654 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.837852 12723 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.838443 15654 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.839082 16739 tablet_copy_client.cc:783] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.839228 16739 tablet_copy_client.cc:647] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.841332 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:53.842056 12856 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
W20240501 19:56:53.844137 12792 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04ccc8addc55441b9ab9d8265b90b03b. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:53.844614 14679 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:53.844976 17015 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.846571 14286 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:53.846977 17015 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:53.848654 17014 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.850283 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04ccc8addc55441b9ab9d8265b90b03b with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:53.850601 15654 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.852579 14680 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.853296 14285 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:53.855855 12453 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 9 to 10, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:53.855906 12723 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.856348 14415 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.858681 14286 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.859275 12446 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.863337 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ccc8addc55441b9ab9d8265b90b03b with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:53.863533 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.863561 16886 tablet_replica.cc:330] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:53.863698 16886 raft_consensus.cc:2229] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:53.863924 16886 raft_consensus.cc:2258] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:53.864042 12593 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:53.864667 16886 ts_tablet_manager.cc:1875] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:53.865506 14415 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.865906 15568 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.866554 16886 ts_tablet_manager.cc:1888] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 19:56:53.866604 16886 log.cc:1196] T 04ccc8addc55441b9ab9d8265b90b03b P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/04ccc8addc55441b9ab9d8265b90b03b
I20240501 19:56:53.867100 14286 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:53.867456 15568 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:53.867884 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.868228 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet 04ccc8addc55441b9ab9d8265b90b03b (table rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]) successfully deleted
I20240501 19:56:53.868948 16739 tablet_copy_client.cc:523] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:53.870291 16739 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:53.870476 15568 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.872145 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d7ea564ea504163a1e17561be9a5130 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:53.872566 14415 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.873909 12449 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:53.874176 14285 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:53.876339 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d7ea564ea504163a1e17561be9a5130 with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:53.876567 16886 tablet_replica.cc:330] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:53.876703 16886 raft_consensus.cc:2229] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:53.876924 16886 raft_consensus.cc:2258] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:53.877723 16886 ts_tablet_manager.cc:1875] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:53.879441 16886 ts_tablet_manager.cc:1888] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:53.879505 16886 log.cc:1196] T 7d7ea564ea504163a1e17561be9a5130 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/7d7ea564ea504163a1e17561be9a5130
I20240501 19:56:53.880025 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet 7d7ea564ea504163a1e17561be9a5130 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
I20240501 19:56:53.883797 17278 tablet_replica.cc:330] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:53.883993 17278 raft_consensus.cc:2229] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:53.884207 17278 raft_consensus.cc:2258] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:53.884860 17278 ts_tablet_manager.cc:1875] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:53.895493 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.903868 17233 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
I20240501 19:56:53.906649 17287 ts_tablet_manager.cc:902] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:53.907537 17287 tablet_copy_client.cc:308] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:53.910100 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 3e84a3865ce64eaebad4df46ec01842d from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:36001)
I20240501 19:56:53.910200 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 3e84a3865ce64eaebad4df46ec01842d from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:36001: session id = 2ca06dad25a645bca71b09741798194a-3e84a3865ce64eaebad4df46ec01842d
I20240501 19:56:53.910966 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.911221 12746 tablet_copy_source_session.cc:205] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.912164 17278 ts_tablet_manager.cc:1888] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 19:56:53.912318 17278 log.cc:1196] T 0fb481d394034a8f85af613beb5c9ec8 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/0fb481d394034a8f85af613beb5c9ec8
I20240501 19:56:53.912904 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 0fb481d394034a8f85af613beb5c9ec8 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
I20240501 19:56:53.916913 17287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e84a3865ce64eaebad4df46ec01842d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.919144 17287 tablet_copy_client.cc:783] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.919291 17287 tablet_copy_client.cc:647] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.931422 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:53.934355 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.938210 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.943735 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:53.949864 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.957653 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:53.958730 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:53.966221 15926 raft_consensus.cc:1050] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:53.966661 15926 consensus_queue.cc:237] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:53.968484 14549 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.969107 12986 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.969633 13712 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.969940 13712 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.971278 14415 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.973600 15472 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.975842 16739 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:53.976020 15472 raft_consensus.cc:1050] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:53.976145 16739 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:53.976311 16739 ts_tablet_manager.cc:1367] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.106s user 0.023s sys 0.000s
I20240501 19:56:53.976466 15472 consensus_queue.cc:237] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.976729 16739 raft_consensus.cc:348] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.976850 16739 raft_consensus.cc:729] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:53.976969 16739 consensus_queue.cc:260] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:53.977358 16739 ts_tablet_manager.cc:1398] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:53.978633 12856 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.979022 15926 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.979133 14286 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.979791 14416 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.980319 12593 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:53.981451 15926 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.981736 15926 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.982860 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-ae2885f7933f462c96db9a99a4eb2090 received from {username='slave'} at 127.5.139.70:36001
I20240501 19:56:53.982934 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2ca06dad25a645bca71b09741798194a-ae2885f7933f462c96db9a99a4eb2090 on tablet ae2885f7933f462c96db9a99a4eb2090 with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:53.984017 16739 ts_tablet_manager.cc:902] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:53.984045 15926 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:53.984403 16739 tablet_copy_client.cc:308] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:53.984798 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 851eeb0ab3364529907d9d09f2a4022e from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:50147)
I20240501 19:56:53.984894 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 851eeb0ab3364529907d9d09f2a4022e from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:50147: session id = 2ca06dad25a645bca71b09741798194a-851eeb0ab3364529907d9d09f2a4022e
I20240501 19:56:53.985867 13007 tablet_copy_source_session.cc:205] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:53.986593 16739 ts_tablet_manager.cc:1875] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:56:53.988720 16739 ts_tablet_manager.cc:1888] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 19:56:53.988901 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851eeb0ab3364529907d9d09f2a4022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:53.990099 17140 raft_consensus.cc:1050] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:53.990175 16739 tablet_copy_client.cc:783] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:53.990301 16739 tablet_copy_client.cc:647] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:53.990504 17140 consensus_queue.cc:237] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:53.991886 14680 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.992085 17327 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.992686 15654 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.994135 14548 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.994166 14547 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.994554 15472 raft_consensus.cc:1050] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:53.994680 15654 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:53.994913 15472 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:53.996713 12857 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:53.996713 12986 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:53.996842 14547 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:53.996837 12453 catalog_manager.cc:5484] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.997530 15472 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:53.997658 12986 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:53.997813 15472 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:53.998278 12857 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:53.999424 15568 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:54.001066 13712 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:54.001263 17287 tablet_copy_client.cc:523] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:54.002534 17287 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:54.003821 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:54.005002 17015 raft_consensus.cc:1050] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:54.005443 17015 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:54.007290 14417 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.007649 12592 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.008249 16064 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.008533 16064 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.008704 15926 raft_consensus.cc:1050] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:54.009130 15926 consensus_queue.cc:237] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:54.009750 12723 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.010149 14285 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.011041 12854 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:54.011436 14549 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:54.011713 13712 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:54.011929 16653 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.012182 16653 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.012192 13712 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:54.015015 14285 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:54.017838 17327 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:54.022209 14415 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.024276 12725 consensus_queue.cc:237] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:54.025605 14546 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:54.026237 14415 raft_consensus.cc:1261] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:54.026563 17322 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.026692 15926 consensus_queue.cc:1035] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.028950 13712 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.030572 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.032152 12445 catalog_manager.cc:5484] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 8 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:54.034806 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 12a50be843fb49ee8a66c9a17d271d61 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:54.037464 14547 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.043161 14415 raft_consensus.cc:2932] T 12a50be843fb49ee8a66c9a17d271d61 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.047111 17327 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.047608 14286 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.049813 12856 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.052223 12453 catalog_manager.cc:5484] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.055374 12593 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.061692 17167 raft_consensus.cc:1050] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:54.062129 17167 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:54.063903 14286 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:54.064332 17158 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:54.065529 12725 consensus_queue.cc:237] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:54.066229 14415 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:54.066612 17158 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:54.067256 14283 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.068004 12595 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.069327 12593 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:54.069815 12856 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.070868 13712 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.071147 13712 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.072451 17158 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:54.072703 17233 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:54.074548 13712 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.078233 13712 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.078963 17158 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.080366 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61bc22cdc6ab45529066fe6f4d14d4c7 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.081238 14283 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.081348 14286 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.081457 12593 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.081849 17233 tablet_bootstrap.cc:492] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:54.082046 17233 ts_tablet_manager.cc:1367] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.178s user 0.025s sys 0.003s
I20240501 19:56:54.082499 17233 raft_consensus.cc:348] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:54.082628 17233 raft_consensus.cc:729] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:54.082743 17233 consensus_queue.cc:260] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:54.083125 17233 ts_tablet_manager.cc:1398] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:54.083699 12593 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.083858 12449 catalog_manager.cc:5484] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.086946 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet 61bc22cdc6ab45529066fe6f4d14d4c7 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca06dad25a645bca71b09741798194a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51964
I20240501 19:56:54.087482 12453 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.089150 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-e4b15761177f4cffb7aa5e4d16e42ac7 received from {username='slave'} at 127.5.139.72:60977
I20240501 19:56:54.089237 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-e4b15761177f4cffb7aa5e4d16e42ac7 on tablet e4b15761177f4cffb7aa5e4d16e42ac7 with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:54.094993 17287 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:54.095031 12725 consensus_queue.cc:237] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:54.095353 17287 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:54.095552 17287 ts_tablet_manager.cc:1367] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.093s user 0.030s sys 0.000s
I20240501 19:56:54.096026 17287 raft_consensus.cc:348] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.096155 17287 raft_consensus.cc:729] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:54.096268 17287 consensus_queue.cc:260] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.096549 17287 ts_tablet_manager.cc:1398] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:54.097460 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-3e84a3865ce64eaebad4df46ec01842d received from {username='slave'} at 127.5.139.70:36001
I20240501 19:56:54.097539 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2ca06dad25a645bca71b09741798194a-3e84a3865ce64eaebad4df46ec01842d on tablet 3e84a3865ce64eaebad4df46ec01842d with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:54.098759 12595 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:54.099287 15472 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:54.099938 14286 raft_consensus.cc:1261] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:54.100566 15472 consensus_queue.cc:1035] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:54.104136 13712 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.105998 14286 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.108551 12444 catalog_manager.cc:5484] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.110334 13712 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.110534 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61bc22cdc6ab45529066fe6f4d14d4c7 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.111513 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 61bc22cdc6ab45529066fe6f4d14d4c7 with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:54.112655 14547 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.113250 12856 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.115481 12444 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.116387 12595 raft_consensus.cc:2932] T 61bc22cdc6ab45529066fe6f4d14d4c7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.117396 17365 tablet_replica.cc:330] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:54.117583 17365 raft_consensus.cc:2229] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.118153 17365 raft_consensus.cc:2258] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.122668 17140 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.124914 14680 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.125345 15472 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.126932 12855 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.127760 14285 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.129189 12855 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.130718 12453 catalog_manager.cc:5484] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.133711 14549 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.134068 16653 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.135463 16653 raft_consensus.cc:1050] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:54.135785 16653 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:54.137379 17389 tablet_replica.cc:330] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:54.137529 17389 raft_consensus.cc:2229] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.137687 17365 ts_tablet_manager.cc:1875] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.137758 17389 pending_rounds.cc:62] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: Trying to abort 1 pending ops.
I20240501 19:56:54.137840 17389 pending_rounds.cc:68] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7022974624011304960 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "61bc22cdc6ab45529066fe6f4d14d4c7" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } } }
I20240501 19:56:54.137964 17389 raft_consensus.cc:2894] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240501 19:56:54.138048 17389 raft_consensus.cc:2866] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:54.138151 17389 raft_consensus.cc:2258] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.140857 17389 ts_tablet_manager.cc:1875] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.141079 12986 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.142819 14549 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.143339 17389 ts_tablet_manager.cc:1888] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:54.143404 17389 log.cc:1196] T 61bc22cdc6ab45529066fe6f4d14d4c7 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/61bc22cdc6ab45529066fe6f4d14d4c7
I20240501 19:56:54.144068 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 61bc22cdc6ab45529066fe6f4d14d4c7 (table rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]) successfully deleted
I20240501 19:56:54.144546 14414 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.146472 12592 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.146718 14417 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.147493 14414 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.148885 17387 tablet_replica.cc:330] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: stopping tablet replica
I20240501 19:56:54.149531 17387 raft_consensus.cc:2229] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Raft consensus shutting down.
I20240501 19:56:54.149681 17387 pending_rounds.cc:62] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Trying to abort 1 pending ops.
I20240501 19:56:54.149750 17387 pending_rounds.cc:68] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022974623646466048 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "61bc22cdc6ab45529066fe6f4d14d4c7" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } } }
I20240501 19:56:54.149938 17387 raft_consensus.cc:2894] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 19:56:54.150031 17387 raft_consensus.cc:2866] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:54.150135 17387 raft_consensus.cc:2258] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Raft consensus is shut down!
I20240501 19:56:54.150452 16064 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.151312 17365 ts_tablet_manager.cc:1888] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:54.151439 17365 log.cc:1196] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/12a50be843fb49ee8a66c9a17d271d61
I20240501 19:56:54.152097 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 12a50be843fb49ee8a66c9a17d271d61 (table rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]) successfully deleted
I20240501 19:56:54.153719 12704 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240501 19:56:54.156054 12445 catalog_manager.cc:5484] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 7 to 8, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.156203 12453 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.159651 12595 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.160228 16064 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.160401 12592 consensus_queue.cc:237] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:54.161931 14548 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:54.162356 15568 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.162555 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:54.164350 17405 ts_tablet_manager.cc:902] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:54.164858 14679 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:54.164960 17405 tablet_copy_client.cc:308] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:54.165338 15568 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.165486 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet 955edb7c12fa4b32ac58827f131931e3 from peer 1d49e80be3d04fb99b711c62d61bac09 ({username='slave'} at 127.5.139.71:49353)
I20240501 19:56:54.165566 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet 955edb7c12fa4b32ac58827f131931e3 from peer 1d49e80be3d04fb99b711c62d61bac09 at {username='slave'} at 127.5.139.71:49353: session id = 1d49e80be3d04fb99b711c62d61bac09-955edb7c12fa4b32ac58827f131931e3
I20240501 19:56:54.166498 12614 tablet_copy_source_session.cc:205] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:54.167255 15568 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.167445 17405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 955edb7c12fa4b32ac58827f131931e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.169440 14679 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.169711 12453 catalog_manager.cc:5484] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.169924 17405 tablet_copy_client.cc:783] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:54.170060 17405 tablet_copy_client.cc:647] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:54.171569 16739 tablet_copy_client.cc:523] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:54.173102 16739 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:54.174314 12725 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.174764 12723 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.175940 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:54.176163 14284 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.176432 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet d940b16d07d54945abe4a10a11dd6a2a with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:54.176965 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d940b16d07d54945abe4a10a11dd6a2a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.178054 12592 consensus_queue.cc:237] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:54.178419 16064 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.179454 17015 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.179697 14284 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.182524 17389 tablet_replica.cc:330] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:54.182713 17389 raft_consensus.cc:2229] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.182940 17389 raft_consensus.cc:2258] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 19:56:54.182941 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db240ec5f8b2444fb6114ad814f3cdb7. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:54.183375 12723 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.183630 14680 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.184079 15569 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.184425 15569 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.187343 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.188773 17015 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.192754 14284 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.194638 12725 consensus_queue.cc:237] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:54.195899 14549 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:54.196499 12448 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.198165 12725 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.198899 15926 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.199412 14284 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:54.199890 15926 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.201719 14414 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.201941 15926 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.203761 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.206020 17014 raft_consensus.cc:1050] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:54.206439 14549 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.206653 17014 consensus_queue.cc:237] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:54.206900 12855 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:54.207935 14397 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.208317 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.208762 12988 raft_consensus.cc:1261] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:54.209206 17039 consensus_queue.cc:1035] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:54.211652 14284 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.214489 12448 catalog_manager.cc:5484] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.214901 14414 raft_consensus.cc:1203] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 19:56:54.215909 12726 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:54.215898 12725 raft_consensus.cc:1261] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:54.217664 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:54.217926 16064 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:54.218199 16064 consensus_queue.cc:1035] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:54.219118 15569 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:54.221897 12723 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:54.223496 12445 catalog_manager.cc:5484] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 6 to 7, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.226923 14679 raft_consensus.cc:1261] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:54.227941 14414 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:54.229180 16653 consensus_queue.cc:1035] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:54.231166 12988 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:54.232370 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.232878 14284 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:54.232903 16653 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:54.233147 14285 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 19:56:54.233721 17015 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:54.233846 16296 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:54.235571 17039 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.236346 16296 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.237423 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 557994658e2c4c8bbc1eeacdab958ddd with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.238065 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 706878734216445f8422e422228d4230 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.238131 14285 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.238430 14414 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 19:56:54.238837 17157 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:54.239415 14286 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.242225 14414 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.242641 12445 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.242920 12445 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 11 to 12, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.245299 12575 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:54.251498 14548 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.253798 12575 tablet_service.cc:1514] Processing DeleteTablet for tablet 557994658e2c4c8bbc1eeacdab958ddd with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:54.254014 17278 tablet_replica.cc:330] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:54.254045 12574 tablet_service.cc:1514] Processing DeleteTablet for tablet 706878734216445f8422e422228d4230 with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:54.254168 17278 raft_consensus.cc:2229] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.254328 17278 pending_rounds.cc:62] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Trying to abort 1 pending ops.
I20240501 19:56:54.254395 17278 pending_rounds.cc:68] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7022974624299552768 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "557994658e2c4c8bbc1eeacdab958ddd" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } } }
I20240501 19:56:54.254586 17278 raft_consensus.cc:2894] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240501 19:56:54.254652 17278 raft_consensus.cc:2866] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:54.254698 17278 raft_consensus.cc:2258] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.254765 14414 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
W20240501 19:56:54.261323 17351 consensus_queue.cc:1159] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" status { last_received { term: 1 index: 7 } last_committed_idx: 6 last_received_current_leader { term: 1 index: 7 } }
I20240501 19:56:54.261778 12855 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:54.266360 14286 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:54.266784 17387 ts_tablet_manager.cc:1875] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.266784 17278 ts_tablet_manager.cc:1875] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.268844 17278 ts_tablet_manager.cc:1888] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 19:56:54.268844 17387 ts_tablet_manager.cc:1888] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 19:56:54.268913 17387 log.cc:1196] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 2ca06dad25a645bca71b09741798194a: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-5/wal/wals/61bc22cdc6ab45529066fe6f4d14d4c7
I20240501 19:56:54.268913 17278 log.cc:1196] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/557994658e2c4c8bbc1eeacdab958ddd
I20240501 19:56:54.268904 16653 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.269642 17278 tablet_replica.cc:330] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:54.269862 17278 raft_consensus.cc:2229] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.270159 17278 raft_consensus.cc:2258] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.270151 12439 catalog_manager.cc:4830] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): tablet 61bc22cdc6ab45529066fe6f4d14d4c7 (table rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]) successfully deleted
I20240501 19:56:54.270704 14414 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:54.270869 17278 ts_tablet_manager.cc:1875] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.271180 16653 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.272861 17278 ts_tablet_manager.cc:1888] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 19:56:54.272976 17278 log.cc:1196] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/706878734216445f8422e422228d4230
I20240501 19:56:54.273577 14679 raft_consensus.cc:1203] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 19:56:54.274653 16739 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:54.275200 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 557994658e2c4c8bbc1eeacdab958ddd (table rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]) successfully deleted
I20240501 19:56:54.275367 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 706878734216445f8422e422228d4230 (table rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]) successfully deleted
I20240501 19:56:54.275864 16739 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:54.276607 16739 ts_tablet_manager.cc:1367] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.104s user 0.028s sys 0.004s
I20240501 19:56:54.278944 16739 raft_consensus.cc:348] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.279176 16739 raft_consensus.cc:729] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:54.279331 16739 consensus_queue.cc:260] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.279878 16064 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.280965 17389 ts_tablet_manager.cc:1875] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.281523 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 557994658e2c4c8bbc1eeacdab958ddd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.282990 17389 ts_tablet_manager.cc:1888] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:54.283089 17389 log.cc:1196] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/d940b16d07d54945abe4a10a11dd6a2a
I20240501 19:56:54.283509 17389 tablet_replica.cc:330] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:54.283836 17389 raft_consensus.cc:2229] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.284020 17389 raft_consensus.cc:2258] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.284087 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet d940b16d07d54945abe4a10a11dd6a2a (table rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]) successfully deleted
I20240501 19:56:54.284859 16739 ts_tablet_manager.cc:1398] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 19:56:54.284924 14286 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.285313 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-851eeb0ab3364529907d9d09f2a4022e received from {username='slave'} at 127.5.139.70:50147
I20240501 19:56:54.285395 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 2ca06dad25a645bca71b09741798194a-851eeb0ab3364529907d9d09f2a4022e on tablet 851eeb0ab3364529907d9d09f2a4022e with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:54.286252 17389 ts_tablet_manager.cc:1875] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.288163 17389 ts_tablet_manager.cc:1888] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:54.288237 17389 log.cc:1196] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/800d5f43cd1b4fb6bd8bbcac262aeb80
I20240501 19:56:54.288712 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 (table rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]) successfully deleted
I20240501 19:56:54.289129 12453 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.289806 14417 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:54.295044 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet 557994658e2c4c8bbc1eeacdab958ddd with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:54.295420 16886 tablet_replica.cc:330] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:54.295570 16886 raft_consensus.cc:2229] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.295737 16886 pending_rounds.cc:62] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: Trying to abort 1 pending ops.
I20240501 19:56:54.295768 16886 pending_rounds.cc:68] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: Aborting op as it isn't in flight: id { term: 1 index: 8 } timestamp: 7022974624590491648 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "557994658e2c4c8bbc1eeacdab958ddd" old_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } } new_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } } }
I20240501 19:56:54.295902 16886 raft_consensus.cc:2894] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Aborting config change with OpId 1.8: Aborted: Op aborted
I20240501 19:56:54.295985 16886 raft_consensus.cc:2866] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:54.296031 16886 raft_consensus.cc:2258] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.296778 16653 raft_consensus.cc:2932] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.297652 14680 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 19:56:54.297785 12723 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60934
I20240501 19:56:54.297924 12723 raft_consensus.cc:595] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:54.297976 16886 ts_tablet_manager.cc:1875] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.300211 16886 ts_tablet_manager.cc:1888] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:54.300279 16886 log.cc:1196] T 557994658e2c4c8bbc1eeacdab958ddd P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/557994658e2c4c8bbc1eeacdab958ddd
I20240501 19:56:54.301156 12988 raft_consensus.cc:2932] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.301772 14680 raft_consensus.cc:2932] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.304497 12451 catalog_manager.cc:5484] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 4 to 5, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.306394 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet 557994658e2c4c8bbc1eeacdab958ddd (table rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]) successfully deleted
I20240501 19:56:54.309234 12855 consensus_queue.cc:237] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:54.310616 14680 raft_consensus.cc:1261] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.311218 17014 consensus_queue.cc:1035] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.312454 12988 raft_consensus.cc:1261] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.312876 17015 consensus_queue.cc:1035] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.313153 17405 tablet_copy_client.cc:523] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:54.313724 17015 raft_consensus.cc:2932] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.314597 17405 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap starting.
I20240501 19:56:54.315037 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d75bd8f836b947248ee124f456c7d721 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.315770 14680 raft_consensus.cc:2932] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.317847 12725 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60960
I20240501 19:56:54.317971 12725 raft_consensus.cc:595] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:54.319468 12453 catalog_manager.cc:5484] T d75bd8f836b947248ee124f456c7d721 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.319762 12988 raft_consensus.cc:2932] T d75bd8f836b947248ee124f456c7d721 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.322968 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet d75bd8f836b947248ee124f456c7d721 with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:54.323197 16886 tablet_replica.cc:330] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:54.323325 16886 raft_consensus.cc:2229] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.323451 16886 pending_rounds.cc:62] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: Trying to abort 1 pending ops.
I20240501 19:56:54.323482 16886 pending_rounds.cc:68] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022974624588500992 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "d75bd8f836b947248ee124f456c7d721" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } } }
I20240501 19:56:54.323601 16886 raft_consensus.cc:2894] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 19:56:54.323657 16886 raft_consensus.cc:2866] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 19:56:54.323697 16886 raft_consensus.cc:2258] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.324460 16886 ts_tablet_manager.cc:1875] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.325196 17322 raft_consensus.cc:979] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: : Instructing follower 2ca06dad25a645bca71b09741798194a to start an election
I20240501 19:56:54.325281 17322 raft_consensus.cc:1067] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Signalling peer 2ca06dad25a645bca71b09741798194a to start an election
I20240501 19:56:54.326112 14417 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7"
dest_uuid: "2ca06dad25a645bca71b09741798194a"
from {username='slave'} at 127.5.139.66:56845
I20240501 19:56:54.326252 14417 raft_consensus.cc:482] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:54.326344 14417 raft_consensus.cc:3037] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.326673 16886 ts_tablet_manager.cc:1888] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:54.326737 16886 log.cc:1196] T d75bd8f836b947248ee124f456c7d721 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/d75bd8f836b947248ee124f456c7d721
I20240501 19:56:54.327311 14417 raft_consensus.cc:504] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:54.327351 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet d75bd8f836b947248ee124f456c7d721 (table rebalance_test_table_3 [id=5f1f9662c1cf49858a630b67ffed42c4]) successfully deleted
I20240501 19:56:54.328359 14417 leader_election.cc:288] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [CANDIDATE]: Term 2 election: Requested vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 19:56:54.328581 12595 consensus_queue.cc:237] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 19:56:54.329355 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7" candidate_uuid: "2ca06dad25a645bca71b09741798194a" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:54.329468 12725 raft_consensus.cc:3032] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:54.329514 12725 raft_consensus.cc:729] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:54.329663 12725 consensus_queue.cc:260] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:54.329849 12725 raft_consensus.cc:3037] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.330619 12855 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.330722 12725 raft_consensus.cc:2454] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca06dad25a645bca71b09741798194a in term 2.
I20240501 19:56:54.331097 14350 leader_election.cc:302] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ca06dad25a645bca71b09741798194a, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:54.331223 15568 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.331940 16641 raft_consensus.cc:2781] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:54.332109 16641 raft_consensus.cc:686] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 2 LEADER]: Becoming Leader. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Running, Role: LEADER
I20240501 19:56:54.332334 16641 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:54.333101 14680 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.333516 15568 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:56:54.334416 12528 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): Couldn't send request to peer 58f750e4d9574c9ebfd3d9e57eefd411. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:54.335211 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7" candidate_uuid: "2ca06dad25a645bca71b09741798194a" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "1d49e80be3d04fb99b711c62d61bac09"
I20240501 19:56:54.335366 14548 raft_consensus.cc:3037] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.335290 12449 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a reported cstate change: term changed from 1 to 2, leader changed from 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) to 2ca06dad25a645bca71b09741798194a (127.5.139.70). New cstate: current_term: 2 leader_uuid: "2ca06dad25a645bca71b09741798194a" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.336634 14548 raft_consensus.cc:2454] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ca06dad25a645bca71b09741798194a in term 2.
I20240501 19:56:54.338021 16739 ts_tablet_manager.cc:902] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:54.338529 16739 tablet_copy_client.cc:308] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:54.338964 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 04ccc8addc55441b9ab9d8265b90b03b from peer 2ca06dad25a645bca71b09741798194a ({username='slave'} at 127.5.139.70:54927)
I20240501 19:56:54.339061 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 04ccc8addc55441b9ab9d8265b90b03b from peer 2ca06dad25a645bca71b09741798194a at {username='slave'} at 127.5.139.70:54927: session id = 2ca06dad25a645bca71b09741798194a-04ccc8addc55441b9ab9d8265b90b03b
I20240501 19:56:54.339901 12877 tablet_copy_source_session.cc:205] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:54.339761 15568 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 19:56:54.340829 16739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ccc8addc55441b9ab9d8265b90b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.342336 12855 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 19:56:54.342473 12449 catalog_manager.cc:5484] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 6 to 7, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.343119 16739 tablet_copy_client.cc:783] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:54.343267 16739 tablet_copy_client.cc:647] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:54.344451 16739 tablet_copy_client.cc:523] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:54.345678 16739 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: Bootstrap starting.
I20240501 19:56:54.350665 14680 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 19:56:54.374925 12988 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ce6add2f19a649b3a12497951047705c"
dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54734
I20240501 19:56:54.375087 12988 raft_consensus.cc:595] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:54.379006 17405 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:54.379423 17405 tablet_bootstrap.cc:492] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: Bootstrap complete.
I20240501 19:56:54.379755 17405 ts_tablet_manager.cc:1367] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: Time spent bootstrapping tablet: real 0.065s user 0.030s sys 0.000s
I20240501 19:56:54.380323 17405 raft_consensus.cc:348] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:54.380523 14417 raft_consensus.cc:1203] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 19:56:54.380556 17405 raft_consensus.cc:729] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Initialized, Role: LEARNER
I20240501 19:56:54.380775 17405 consensus_queue.cc:260] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
I20240501 19:56:54.381196 17405 ts_tablet_manager.cc:1398] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:54.383898 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session 1d49e80be3d04fb99b711c62d61bac09-955edb7c12fa4b32ac58827f131931e3 received from {username='slave'} at 127.5.139.71:49353
I20240501 19:56:54.383978 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session 1d49e80be3d04fb99b711c62d61bac09-955edb7c12fa4b32ac58827f131931e3 on tablet 955edb7c12fa4b32ac58827f131931e3 with peer 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:54.393371 16739 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:54.393651 16739 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: Bootstrap complete.
I20240501 19:56:54.393832 16739 ts_tablet_manager.cc:1367] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: Time spent bootstrapping tablet: real 0.048s user 0.029s sys 0.000s
I20240501 19:56:54.394476 16739 raft_consensus.cc:348] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.394637 16739 raft_consensus.cc:729] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2ca06dad25a645bca71b09741798194a, State: Initialized, Role: LEARNER
I20240501 19:56:54.394745 16739 consensus_queue.cc:260] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.395010 16739 ts_tablet_manager.cc:1398] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:54.395484 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session 2ca06dad25a645bca71b09741798194a-04ccc8addc55441b9ab9d8265b90b03b received from {username='slave'} at 127.5.139.70:54927
I20240501 19:56:54.395570 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session 2ca06dad25a645bca71b09741798194a-04ccc8addc55441b9ab9d8265b90b03b on tablet 04ccc8addc55441b9ab9d8265b90b03b with peer 2ca06dad25a645bca71b09741798194a
I20240501 19:56:54.397822 17322 raft_consensus.cc:979] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: : Instructing follower ebbbd9ac07a94e8dbbae6def0a2df415 to start an election
I20240501 19:56:54.397917 17322 raft_consensus.cc:1067] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Signalling peer ebbbd9ac07a94e8dbbae6def0a2df415 to start an election
I20240501 19:56:54.398391 12595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a"
dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
from {username='slave'} at 127.5.139.66:45691
I20240501 19:56:54.398530 12595 raft_consensus.cc:482] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:54.398578 12595 raft_consensus.cc:3037] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.400094 12595 raft_consensus.cc:504] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:54.400133 12725 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60994
I20240501 19:56:54.400238 12725 raft_consensus.cc:595] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:54.400548 12595 leader_election.cc:288] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 2 election: Requested vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 19:56:54.400707 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:54.400797 12725 raft_consensus.cc:3032] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:54.400830 12725 raft_consensus.cc:729] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:54.400945 12725 consensus_queue.cc:260] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:54.401131 12725 raft_consensus.cc:3037] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.401331 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "9069ec6fcdb248479037f2d37e0e851d"
I20240501 19:56:54.401515 14286 raft_consensus.cc:3037] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.402004 12725 raft_consensus.cc:2454] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 2.
I20240501 19:56:54.402357 14286 raft_consensus.cc:2454] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 2.
I20240501 19:56:54.402631 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e4d79f5e11b4923acb2ce543884b54a" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1d49e80be3d04fb99b711c62d61bac09"
I20240501 19:56:54.402666 12528 leader_election.cc:302] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, 9069ec6fcdb248479037f2d37e0e851d, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:54.402730 14548 raft_consensus.cc:3037] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.402825 15654 raft_consensus.cc:2781] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:54.402891 15654 raft_consensus.cc:686] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEADER]: Becoming Leader. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:54.403019 15654 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:54.403412 14548 raft_consensus.cc:2454] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 2.
I20240501 19:56:54.404752 12449 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: term changed from 1 to 2, leader changed from 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) to ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65). New cstate: current_term: 2 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.455943 14417 raft_consensus.cc:1203] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 19:56:54.467453 17157 raft_consensus.cc:979] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: : Instructing follower ebbbd9ac07a94e8dbbae6def0a2df415 to start an election
I20240501 19:56:54.467615 17157 raft_consensus.cc:1067] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Signalling peer ebbbd9ac07a94e8dbbae6def0a2df415 to start an election
I20240501 19:56:54.468197 12595 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ce6add2f19a649b3a12497951047705c"
dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
from {username='slave'} at 127.5.139.68:57561
I20240501 19:56:54.468333 12595 raft_consensus.cc:482] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:54.468410 12595 raft_consensus.cc:3037] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.469456 12595 raft_consensus.cc:504] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:54.469918 12595 leader_election.cc:288] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 2 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
I20240501 19:56:54.470916 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6add2f19a649b3a12497951047705c" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:54.471041 12988 raft_consensus.cc:3032] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:54.471086 12988 raft_consensus.cc:729] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:54.471132 14680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6add2f19a649b3a12497951047705c" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c6d2c7096e404ea9a6f73b123d74c622"
I20240501 19:56:54.471242 12988 consensus_queue.cc:260] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:54.471294 14680 raft_consensus.cc:3037] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.471421 12988 raft_consensus.cc:3037] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.472347 12988 raft_consensus.cc:2454] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 2.
I20240501 19:56:54.472354 14680 raft_consensus.cc:2454] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 2.
I20240501 19:56:54.472903 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce6add2f19a649b3a12497951047705c" candidate_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:54.473032 12855 raft_consensus.cc:3037] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.473076 12531 leader_election.cc:302] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, c6d2c7096e404ea9a6f73b123d74c622, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:54.473301 17140 raft_consensus.cc:2781] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:54.473381 17140 raft_consensus.cc:686] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEADER]: Becoming Leader. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:54.473496 17140 consensus_queue.cc:237] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:54.474083 12855 raft_consensus.cc:2454] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ebbbd9ac07a94e8dbbae6def0a2df415 in term 2.
I20240501 19:56:54.475523 12449 catalog_manager.cc:5484] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: term changed from 1 to 2, leader changed from 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) to ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65). New cstate: current_term: 2 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.478003 12595 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cf6948589837482a9f580731e3b861b8"
dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48866
I20240501 19:56:54.478159 12595 raft_consensus.cc:595] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:54.503602 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.514871 12967 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:54.518879 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.519153 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.524247 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:54.530005 14397 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:54.531260 12574 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:54.558238 12704 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:54.560370 14548 raft_consensus.cc:1203] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 19:56:54.587268 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:54.590857 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.593673 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.604892 12967 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:54.605917 14397 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:54.619951 12725 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:32818
I20240501 19:56:54.620100 12725 raft_consensus.cc:595] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:54.623517 14266 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.629451 12704 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:54.644351 15654 raft_consensus.cc:979] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: : Instructing follower 9069ec6fcdb248479037f2d37e0e851d to start an election
I20240501 19:56:54.644454 15654 raft_consensus.cc:1067] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Signalling peer 9069ec6fcdb248479037f2d37e0e851d to start an election
I20240501 19:56:54.645097 14286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cf6948589837482a9f580731e3b861b8"
dest_uuid: "9069ec6fcdb248479037f2d37e0e851d"
from {username='slave'} at 127.5.139.65:57207
I20240501 19:56:54.645267 14286 raft_consensus.cc:482] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:54.645323 14286 raft_consensus.cc:3037] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.646327 14286 raft_consensus.cc:504] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:54.648092 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "9069ec6fcdb248479037f2d37e0e851d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:54.648212 12855 raft_consensus.cc:3037] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.649178 12855 raft_consensus.cc:2454] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9069ec6fcdb248479037f2d37e0e851d in term 2.
I20240501 19:56:54.649580 14286 leader_election.cc:288] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [CANDIDATE]: Term 2 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933)
I20240501 19:56:54.650171 12595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "9069ec6fcdb248479037f2d37e0e851d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:54.650301 12595 raft_consensus.cc:3032] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:54.650352 12595 raft_consensus.cc:729] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:54.650517 12595 consensus_queue.cc:260] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:54.650719 12595 raft_consensus.cc:3037] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.651651 12595 raft_consensus.cc:2454] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9069ec6fcdb248479037f2d37e0e851d in term 2.
I20240501 19:56:54.654024 14221 leader_election.cc:302] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 9069ec6fcdb248479037f2d37e0e851d, a238fc0c6d844cedb8adccb97d01ae5e, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:54.654243 17386 raft_consensus.cc:2781] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:54.654395 17386 raft_consensus.cc:686] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Becoming Leader. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Running, Role: LEADER
I20240501 19:56:54.654739 17386 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:54.655419 13712 raft_consensus.cc:1050] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:54.655799 13712 consensus_queue.cc:237] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:54.657444 12986 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:54.657403 12451 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: term changed from 1 to 2, leader changed from ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) to 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69). New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.658048 15926 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:54.658751 14417 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:54.658926 12595 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 19:56:54.661454 15472 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:54.661787 15472 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 19:56:54.665910 12574 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:54.673972 14417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "9069ec6fcdb248479037f2d37e0e851d" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "2ca06dad25a645bca71b09741798194a"
I20240501 19:56:54.674132 14417 raft_consensus.cc:3037] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.675124 14417 raft_consensus.cc:2454] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9069ec6fcdb248479037f2d37e0e851d in term 2.
I20240501 19:56:54.677232 15926 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.679561 14417 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.680114 12451 catalog_manager.cc:5484] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:54.682202 17737 ts_tablet_manager.cc:902] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:54.683329 17737 tablet_copy_client.cc:308] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:54.683779 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet db240ec5f8b2444fb6114ad814f3cdb7 from peer 9069ec6fcdb248479037f2d37e0e851d ({username='slave'} at 127.5.139.69:60521)
I20240501 19:56:54.683871 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet db240ec5f8b2444fb6114ad814f3cdb7 from peer 9069ec6fcdb248479037f2d37e0e851d at {username='slave'} at 127.5.139.69:60521: session id = 9069ec6fcdb248479037f2d37e0e851d-db240ec5f8b2444fb6114ad814f3cdb7
I20240501 19:56:54.684721 12614 tablet_copy_source_session.cc:205] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:54.685591 17737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db240ec5f8b2444fb6114ad814f3cdb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.689092 17737 tablet_copy_client.cc:783] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:54.689250 17737 tablet_copy_client.cc:647] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:54.690953 12986 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.702214 12595 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.707418 12725 consensus_queue.cc:237] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:54.709334 12595 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.709371 14417 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.709743 15472 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.710052 15472 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.712829 13712 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.715284 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae2885f7933f462c96db9a99a4eb2090 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.715641 14417 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.718437 12445 catalog_manager.cc:5484] T ae2885f7933f462c96db9a99a4eb2090 P 2ca06dad25a645bca71b09741798194a reported cstate change: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.718420 12595 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:54.720726 17737 tablet_copy_client.cc:523] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:54.722134 17737 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap starting.
I20240501 19:56:54.727085 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet ae2885f7933f462c96db9a99a4eb2090 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:54.727429 17767 tablet_replica.cc:330] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:54.727566 17767 raft_consensus.cc:2229] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.727739 17767 raft_consensus.cc:2258] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.728447 17767 ts_tablet_manager.cc:1875] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.730769 17767 ts_tablet_manager.cc:1888] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:54.730919 17767 log.cc:1196] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/ae2885f7933f462c96db9a99a4eb2090
I20240501 19:56:54.731600 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet ae2885f7933f462c96db9a99a4eb2090 (table rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]) successfully deleted
I20240501 19:56:54.732218 12592 consensus_queue.cc:237] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.733956 12986 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.734414 15569 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.736068 14680 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.736621 15569 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:56:54.737129 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22b272a8582d4060a6a527ac4fce9ae5. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:54.738710 15568 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.739106 12986 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.739229 14680 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.742200 14548 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer 2ca06dad25a645bca71b09741798194a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:54.742745 17472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:54.743273 12444 catalog_manager.cc:5484] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.746591 12725 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2ca06dad25a645bca71b09741798194a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:54.747004 16641 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:54.753188 14680 raft_consensus.cc:3037] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Advancing to term 2
I20240501 19:56:54.754230 14680 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 LEARNER]: Refusing update from remote peer 2ca06dad25a645bca71b09741798194a: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:54.754923 17773 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:54.778460 17773 raft_consensus.cc:1050] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:54.779347 17773 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 10, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:54.781234 12725 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2ca06dad25a645bca71b09741798194a: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:54.781327 14548 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer 2ca06dad25a645bca71b09741798194a: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:54.781737 17472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:54.782177 17472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:54.782868 14680 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 LEARNER]: Refusing update from remote peer 2ca06dad25a645bca71b09741798194a: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:54.783303 17472 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:54.785778 16641 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 10 to 12, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.793942 12725 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.794931 14548 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.795841 12448 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a reported cstate change: config changed from index 10 to 12, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2ca06dad25a645bca71b09741798194a" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:54.796660 14680 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.798110 14417 raft_consensus.cc:1203] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 19:56:54.803802 14417 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:54.805483 14680 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Refusing update from remote peer 2ca06dad25a645bca71b09741798194a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:54.805682 14266 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:54.805711 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.805929 17773 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:54.813287 17737 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:54.813570 17737 tablet_bootstrap.cc:492] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: Bootstrap complete.
I20240501 19:56:54.813735 17737 ts_tablet_manager.cc:1367] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: Time spent bootstrapping tablet: real 0.092s user 0.027s sys 0.000s
I20240501 19:56:54.814230 17737 raft_consensus.cc:348] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:54.814354 17737 raft_consensus.cc:729] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Initialized, Role: LEARNER
I20240501 19:56:54.814491 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:54.814505 17773 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.816125 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4b15761177f4cffb7aa5e4d16e42ac7 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.817180 14548 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer 2ca06dad25a645bca71b09741798194a: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:54.817610 16641 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:54.817626 12449 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P 2ca06dad25a645bca71b09741798194a reported cstate change: config changed from index 12 to 13, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 2 leader_uuid: "2ca06dad25a645bca71b09741798194a" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:54.817678 14680 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.818004 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:54.818682 14548 raft_consensus.cc:2932] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.821137 17737 consensus_queue.cc:260] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:54.822371 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b15761177f4cffb7aa5e4d16e42ac7 with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:54.822621 16886 tablet_replica.cc:330] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:54.822788 16886 raft_consensus.cc:2229] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.822942 16886 raft_consensus.cc:2258] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.823333 12574 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:54.823743 16886 ts_tablet_manager.cc:1875] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.826077 16886 ts_tablet_manager.cc:1888] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 19:56:54.826144 16886 log.cc:1196] T e4b15761177f4cffb7aa5e4d16e42ac7 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/e4b15761177f4cffb7aa5e4d16e42ac7
I20240501 19:56:54.826704 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet e4b15761177f4cffb7aa5e4d16e42ac7 (table rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]) successfully deleted
I20240501 19:56:54.831617 14397 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:54.847863 12704 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:54.849303 17737 ts_tablet_manager.cc:1398] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d: Time spent starting tablet: real 0.036s user 0.002s sys 0.000s
I20240501 19:56:54.851022 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session 9069ec6fcdb248479037f2d37e0e851d-db240ec5f8b2444fb6114ad814f3cdb7 received from {username='slave'} at 127.5.139.69:60521
I20240501 19:56:54.851123 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session 9069ec6fcdb248479037f2d37e0e851d-db240ec5f8b2444fb6114ad814f3cdb7 on tablet db240ec5f8b2444fb6114ad814f3cdb7 with peer 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:54.856842 12967 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:54.885540 12725 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:54.885905 17140 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.888567 15472 raft_consensus.cc:979] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: : Instructing follower 9069ec6fcdb248479037f2d37e0e851d to start an election
I20240501 19:56:54.888655 15472 raft_consensus.cc:1067] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Signalling peer 9069ec6fcdb248479037f2d37e0e851d to start an election
I20240501 19:56:54.889122 14286 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d"
dest_uuid: "9069ec6fcdb248479037f2d37e0e851d"
from {username='slave'} at 127.5.139.66:58123
I20240501 19:56:54.889245 14286 raft_consensus.cc:482] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:54.889300 14286 raft_consensus.cc:3037] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.890341 14286 raft_consensus.cc:504] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.890772 14286 leader_election.cc:288] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [CANDIDATE]: Term 2 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:54.891443 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d" candidate_uuid: "9069ec6fcdb248479037f2d37e0e851d" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:54.891558 12855 raft_consensus.cc:3037] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.892604 12855 raft_consensus.cc:2454] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9069ec6fcdb248479037f2d37e0e851d in term 2.
I20240501 19:56:54.893208 14219 leader_election.cc:302] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9069ec6fcdb248479037f2d37e0e851d, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:54.893410 17722 raft_consensus.cc:2781] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:54.893489 17722 raft_consensus.cc:686] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Becoming Leader. State: Replica: 9069ec6fcdb248479037f2d37e0e851d, State: Running, Role: LEADER
I20240501 19:56:54.893635 17722 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.894894 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d" candidate_uuid: "9069ec6fcdb248479037f2d37e0e851d" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:54.895025 12725 raft_consensus.cc:3032] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:54.895085 12725 raft_consensus.cc:729] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:54.895220 12725 consensus_queue.cc:260] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.895411 12725 raft_consensus.cc:3037] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:54.895615 12451 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: term changed from 1 to 2, leader changed from 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) to 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69). New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.896358 12725 raft_consensus.cc:2454] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9069ec6fcdb248479037f2d37e0e851d in term 2.
I20240501 19:56:54.908512 14680 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:54.909569 17140 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.925192 17867 ts_tablet_manager.cc:902] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:54.925954 17867 tablet_copy_client.cc:308] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:54.926508 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet 544caa38514643d5b81cd6ed85e0bae7 from peer 58f750e4d9574c9ebfd3d9e57eefd411 ({username='slave'} at 127.5.139.66:45691)
I20240501 19:56:54.926620 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet 544caa38514643d5b81cd6ed85e0bae7 from peer 58f750e4d9574c9ebfd3d9e57eefd411 at {username='slave'} at 127.5.139.66:45691: session id = 58f750e4d9574c9ebfd3d9e57eefd411-544caa38514643d5b81cd6ed85e0bae7
I20240501 19:56:54.927556 12614 tablet_copy_source_session.cc:205] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:54.928494 17867 ts_tablet_manager.cc:1875] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:56:54.928887 12725 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:54.929747 17167 raft_consensus.cc:1050] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:54.930174 17167 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:54.931088 12855 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.931502 13712 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.932065 14548 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:54.932447 17157 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 19:56:54.933754 12661 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ca3285abe3654723a6922672391e9423. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:54.934973 12854 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:54.935195 12986 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.935649 14417 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:54.936112 17157 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:54.936934 17322 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.937114 14549 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 19:56:54.937561 17322 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.939498 13712 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.937563 17867 ts_tablet_manager.cc:1888] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 19:56:54.941222 17867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 544caa38514643d5b81cd6ed85e0bae7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:54.941720 12856 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.942972 17867 tablet_copy_client.cc:783] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:54.944012 15508 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:54.944888 12451 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.945958 17867 tablet_copy_client.cc:647] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:54.947796 17867 tablet_copy_client.cc:523] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:54.948591 14549 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.949075 12725 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:54.950984 12854 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 19:56:54.952729 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 19:56:54.953290 17867 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:54.955555 12857 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
W20240501 19:56:54.956048 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): Couldn't send request to peer 2d52575d9e1b40dca0fc553d7295eb74. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:54.956081 15569 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 19:56:54.957605 12857 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:54.957764 12986 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } } }
I20240501 19:56:54.959311 12725 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
W20240501 19:56:54.959465 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 557994658e2c4c8bbc1eeacdab958ddd. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:54.959947 14414 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:54.960311 17015 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:54.960860 14286 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:54.961266 14547 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:54.961685 17140 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.961748 17015 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:54.963732 15926 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:54.964042 17015 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:54.965978 14286 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:54.966384 12854 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:54.969492 12444 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 9 to 10, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:54.969841 12449 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 14 to 15, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:54.973237 14414 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 19:56:54.979995 12857 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.980444 17322 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.980512 12592 consensus_queue.cc:237] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:54.980705 12986 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240501 19:56:54.981376 14547 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.982399 17322 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.982764 14680 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:56:54.983876 17140 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.985177 12986 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:54.985566 17322 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.985543 17140 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.987020 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce6add2f19a649b3a12497951047705c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.987324 14680 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.988168 15926 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.988371 14549 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 19:56:54.988611 12857 raft_consensus.cc:1261] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240501 19:56:54.989143 15568 consensus_queue.cc:1035] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 19:56:54.989748 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca3285abe3654723a6922672391e9423 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:54.990303 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 19:56:54.990363 12444 catalog_manager.cc:5484] T ce6add2f19a649b3a12497951047705c P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 5 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 2 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.991034 14549 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.991297 12445 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:54.992362 14286 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:54.992486 12855 raft_consensus.cc:2932] T ce6add2f19a649b3a12497951047705c P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:54.992792 15569 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:54.993290 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet ce6add2f19a649b3a12497951047705c with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:54.993499 17767 tablet_replica.cc:330] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:54.993626 17767 raft_consensus.cc:2229] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:54.993717 17767 raft_consensus.cc:2258] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:54.994177 14397 tablet_service.cc:1514] Processing DeleteTablet for tablet ca3285abe3654723a6922672391e9423 with delete_type TABLET_DATA_TOMBSTONED (TS 2ca06dad25a645bca71b09741798194a not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51964
I20240501 19:56:54.994331 17767 ts_tablet_manager.cc:1875] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:54.994520 12857 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:54.994887 14547 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:54.996443 17767 ts_tablet_manager.cc:1888] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 19:56:54.996512 17767 log.cc:1196] T ce6add2f19a649b3a12497951047705c P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/ce6add2f19a649b3a12497951047705c
I20240501 19:56:54.997386 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet ce6add2f19a649b3a12497951047705c (table rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]) successfully deleted
I20240501 19:56:54.998936 12986 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
W20240501 19:56:55.014828 12439 catalog_manager.cc:4797] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): delete failed for tablet ca3285abe3654723a6922672391e9423 because the tablet was not found. No further retry: Not found: Tablet not found: ca3285abe3654723a6922672391e9423
I20240501 19:56:55.016453 15508 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.016860 14548 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.018863 14417 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.019924 12857 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.021719 12449 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:55.032794 12986 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:55.034626 14548 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 19:56:55.034981 17157 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:55.035666 14417 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 19:56:55.036062 17157 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:55.041262 17157 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.043972 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 851eeb0ab3364529907d9d09f2a4022e with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:55.044175 12448 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 11 to 12, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:55.044322 12986 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c7ddb6d79d384d39a709b5824f936d53"
dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54852
I20240501 19:56:55.044430 12986 raft_consensus.cc:595] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:55.044813 14548 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.047317 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 851eeb0ab3364529907d9d09f2a4022e with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:55.049660 14417 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.060854 12592 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:55.061707 17722 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:55.062636 14417 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 2ca06dad25a645bca71b09741798194a [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:55.063341 17386 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:55.066257 17877 raft_consensus.cc:979] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: : Instructing follower 58f750e4d9574c9ebfd3d9e57eefd411 to start an election
I20240501 19:56:55.066339 17877 raft_consensus.cc:1067] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Signalling peer 58f750e4d9574c9ebfd3d9e57eefd411 to start an election
I20240501 19:56:55.066821 12725 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c7ddb6d79d384d39a709b5824f936d53"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
from {username='slave'} at 127.5.139.68:58755
I20240501 19:56:55.066951 12725 raft_consensus.cc:482] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:55.067005 12725 raft_consensus.cc:3037] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.067983 12725 raft_consensus.cc:504] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:55.068374 12725 leader_election.cc:288] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 2 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719)
I20240501 19:56:55.068992 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ddb6d79d384d39a709b5824f936d53" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:55.069164 12857 raft_consensus.cc:3037] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.070079 12857 raft_consensus.cc:2454] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 2.
I20240501 19:56:55.070318 14286 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ddb6d79d384d39a709b5824f936d53" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "9069ec6fcdb248479037f2d37e0e851d"
I20240501 19:56:55.070411 14286 raft_consensus.cc:3037] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.071322 14286 raft_consensus.cc:2454] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 2.
I20240501 19:56:55.072044 12659 leader_election.cc:302] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, 9069ec6fcdb248479037f2d37e0e851d, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
I20240501 19:56:55.072219 13712 raft_consensus.cc:2781] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:55.072299 13712 raft_consensus.cc:686] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:55.072454 13712 consensus_queue.cc:237] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:55.074353 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c7ddb6d79d384d39a709b5824f936d53" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:55.074461 12986 raft_consensus.cc:3032] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:55.074498 12986 raft_consensus.cc:729] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:55.074469 12449 catalog_manager.cc:5484] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 1 to 2, leader changed from 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 2 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:55.074640 12986 consensus_queue.cc:260] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:55.074817 12986 raft_consensus.cc:3037] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.075824 12986 raft_consensus.cc:2454] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 2.
I20240501 19:56:55.077302 17935 tablet_replica.cc:330] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:55.077427 17935 raft_consensus.cc:2229] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.079442 17935 raft_consensus.cc:2258] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.079704 12857 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:55.080080 17722 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:55.081133 17935 ts_tablet_manager.cc:1875] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:55.087841 12986 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
W20240501 19:56:55.089663 12921 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): Couldn't send request to peer 58f750e4d9574c9ebfd3d9e57eefd411. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.090533 14680 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:55.091126 17167 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.091168 14286 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:55.091553 17167 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.095788 17867 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:55.096123 17867 tablet_bootstrap.cc:492] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap complete.
I20240501 19:56:55.096300 17867 ts_tablet_manager.cc:1367] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.145s user 0.031s sys 0.000s
I20240501 19:56:55.096769 17867 raft_consensus.cc:348] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 19:56:55.096887 17867 raft_consensus.cc:729] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: LEARNER
I20240501 19:56:55.096999 17867 consensus_queue.cc:260] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 19:56:55.097323 17867 ts_tablet_manager.cc:1398] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.097961 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-544caa38514643d5b81cd6ed85e0bae7 received from {username='slave'} at 127.5.139.66:45691
I20240501 19:56:55.098059 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-544caa38514643d5b81cd6ed85e0bae7 on tablet 544caa38514643d5b81cd6ed85e0bae7 with peer 58f750e4d9574c9ebfd3d9e57eefd411
I20240501 19:56:55.098409 17935 ts_tablet_manager.cc:1888] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 19:56:55.098488 17935 log.cc:1196] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/851eeb0ab3364529907d9d09f2a4022e
I20240501 19:56:55.099673 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 851eeb0ab3364529907d9d09f2a4022e (table rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]) successfully deleted
I20240501 19:56:55.099712 17167 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 19:56:55.102341 14286 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 19:56:55.104555 12445 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 9 to 10, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:55.113468 14680 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 19:56:55.125301 12725 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:55.127019 14548 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 19:56:55.127140 12662 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): Couldn't send request to peer 2d52575d9e1b40dca0fc553d7295eb74. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.127796 14417 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:55.128252 17322 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.128805 17322 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.133193 17322 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.133682 14417 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.137100 14548 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.137341 12449 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 9 to 10, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:55.161275 15569 raft_consensus.cc:1050] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 to VOTER
I20240501 19:56:55.161715 15569 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:55.163476 12986 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:55.164045 17140 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.164405 14680 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:55.165095 14548 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 19:56:55.166254 17140 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.167812 17140 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.172319 17140 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:55.177946 14680 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:55.179986 14548 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:55.182294 12986 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:55.186187 12445 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 9 to 10, 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:55.235162 12967 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:55.247776 14397 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:55.248211 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 19:56:55.262312 14266 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:55.265760 14286 raft_consensus.cc:1203] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:55.269491 17015 raft_consensus.cc:1050] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:55.269937 17015 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:55.271366 12704 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:55.271396 14680 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:55.271932 14286 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:55.271930 14417 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:55.271970 18080 ts_tablet_manager.cc:902] T 22b272a8582d4060a6a527ac4fce9ae5 P 2ca06dad25a645bca71b09741798194a: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:55.272315 16653 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
F20240501 19:56:55.272238 18080 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.queue_time_us, tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.queue_time_us) (tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_cpu_time_us, tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_cpu_time_us) (tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_wall_time_us, tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_wall_time_us) (tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.queue_time_us, tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.queue_time_us) (tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_cpu_time_us, tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_cpu_time_us) (tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_wall_time_us, tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_wall_time_us) (tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.queue_time_us, tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.queue_time_us) (tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.run_cpu_time_us, tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.run_cpu_time_us) (tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.run_wall_time_us, tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.run_wall_time_us) (tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.queue_time_us, tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.queue_time_us) (tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.queue_time_us, tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.queue_time_us) (tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_cpu_time_us, tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_cpu_time_us) (tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_wall_time_us, tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_wall_time_us) (tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.queue_time_us, tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.queue_time_us) (tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_cpu_time_us, tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_cpu_time_us) (tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_wall_time_us, tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_wall_time_us) (tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.queue_time_us, tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.queue_time_us) (tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.run_cpu_time_us, tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.run_cpu_time_us) (tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.run_wall_time_us, tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.run_wall_time_us) (tablet-download-pool-706878734216445f8422e422228d4230.queue_time_us, tablet-download-pool-706878734216445f8422e422228d4230.queue_time_us) (tablet-download-pool-706878734216445f8422e422228d4230.run_cpu_time_us, tablet-download-pool-706878734216445f8422e422228d4230.run_cpu_time_us) (tablet-download-pool-706878734216445f8422e422228d4230.run_wall_time_us, tablet-download-pool-706878734216445f8422e422228d4230.run_wall_time_us) (tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.queue_time_us, tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.queue_time_us) (tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.run_cpu_time_us, tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.run_cpu_time_us) (tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.run_wall_time_us, tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.run_wall_time_us) (tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.queue_time_us, tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.queue_time_us) (tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.run_cpu_time_us, tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.run_cpu_time_us) (tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.run_wall_time_us, tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.run_wall_time_us) (tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.queue_time_us, tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.queue_time_us) (tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.run_cpu_time_us, tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.run_cpu_time_us) (tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.run_wall_time_us, tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.run_wall_time_us) (tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.queue_time_us, tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.queue_time_us) (tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.run_cpu_time_us, tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.run_cpu_time_us) (tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.run_wall_time_us, tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.run_wall_time_us) (tablet-download-pool-ae2885f7933f462c96db9a99a4eb2090.queue_time_us, tablet-download-pool-ae2885f7933f462c96db9a99a4eb2090.queue_time_us) (tablet-download-pool-ae2885f7933f462c96db9a99a4eb2090.run_cpu_time_us, tablet-download-pool-ae2885f7933f462c96db9a99a4eb2090.run_cpu_time_us) (tablet-download-pool-ae2885f7933f462c96db9a99a4eb2090.run_wall_time_us, tablet-download-pool-ae2885f7933f462c96db9a99a4eb2090.run_wall_time_us) (tablet-download-pool-cf6948589837482a9f580731e3b861b8.queue_time_us, tablet-download-pool-cf6948589837482a9f580731e3b861b8.queue_time_us) (tablet-download-pool-cf6948589837482a9f580731e3b861b8.run_cpu_time_us, tablet-download-pool-cf6948589837482a9f580731e3b861b8.run_cpu_time_us) (tablet-download-pool-cf6948589837482a9f580731e3b861b8.run_wall_time_us, tablet-download-pool-cf6948589837482a9f580731e3b861b8.run_wall_time_us) (tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.queue_time_us, tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.queue_time_us) (tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_cpu_time_us, tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_cpu_time_us) (tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_wall_time_us, tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240501 19:56:55.272491 17014 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
*** Check failure stack trace: ***
I20240501 19:56:55.272892 17014 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
@ 0x7f89791f2dcd google::LogMessage::Fail() at ??:0
I20240501 19:56:55.290253 12967 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 19:56:55.309396 14414 raft_consensus.cc:3037] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a [term 1 LEARNER]: Advancing to term 2
I20240501 19:56:55.309940 12704 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 19:56:55.310545 14414 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a [term 2 LEARNER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:55.312180 17015 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.313352 17968 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:55.314476 14417 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.316334 14680 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.317387 12445 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P 2ca06dad25a645bca71b09741798194a reported cstate change: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.320487 14286 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.321581 14266 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:55.324556 12574 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:55.329514 17968 raft_consensus.cc:1050] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
W20240501 19:56:55.329677 17968 raft_consensus.cc:1053] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: Unable to promote non-voter 2ca06dad25a645bca71b09741798194a: Illegal state: Leader has not yet committed an operation in its own term
I20240501 19:56:55.335829 14397 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:55.337913 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f89791f6b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f89791f27cc google::LogMessage::Flush() at ??:0
I20240501 19:56:55.353622 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:55.355183 14660 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 19:56:55.355384 12836 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
@ 0x7f89791f3f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 19:56:55.357873 12574 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
@ 0x7f897a65c044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f897a640f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f897a63eefc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f897d1f177a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f897d1faa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 19:56:55.379112 18128 ts_tablet_manager.cc:902] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 19:56:55.379797 18128 tablet_copy_client.cc:308] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 19:56:55.380251 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 557994658e2c4c8bbc1eeacdab958ddd from peer c6d2c7096e404ea9a6f73b123d74c622 ({username='slave'} at 127.5.139.72:37741)
I20240501 19:56:55.380342 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 557994658e2c4c8bbc1eeacdab958ddd from peer c6d2c7096e404ea9a6f73b123d74c622 at {username='slave'} at 127.5.139.72:37741: session id = c6d2c7096e404ea9a6f73b123d74c622-557994658e2c4c8bbc1eeacdab958ddd
I20240501 19:56:55.381317 12877 tablet_copy_source_session.cc:205] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:55.381366 14529 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7f897d27b76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 19:56:55.389407 18128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557994658e2c4c8bbc1eeacdab958ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.391619 18128 tablet_copy_client.cc:783] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:55.391758 18128 tablet_copy_client.cc:647] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:55.393332 18128 tablet_copy_client.cc:523] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:55.394758 12725 raft_consensus.cc:1203] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:55.395864 18128 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap starting.
@ 0x7f897d28f4f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f897d27a226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
I20240501 19:56:55.415645 18142 ts_tablet_manager.cc:902] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:55.416390 18142 tablet_copy_client.cc:308] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:55.416939 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 8b61e94fc7e541c8aa84957560508a31 from peer 2d52575d9e1b40dca0fc553d7295eb74 ({username='slave'} at 127.5.139.68:58755)
I20240501 19:56:55.417097 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 8b61e94fc7e541c8aa84957560508a31 from peer 2d52575d9e1b40dca0fc553d7295eb74 at {username='slave'} at 127.5.139.68:58755: session id = 2d52575d9e1b40dca0fc553d7295eb74-8b61e94fc7e541c8aa84957560508a31
@ 0x7f897d289c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
I20240501 19:56:55.417865 12725 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:55.417982 12746 tablet_copy_source_session.cc:205] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
@ 0x55de9239375c std::function<>::operator()() at ??:0
I20240501 19:56:55.418815 12857 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:55.419149 17968 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:55.419714 17968 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
@ 0x7f897a643e06 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 19:56:55.421641 18142 ts_tablet_manager.cc:1875] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:56:55.424495 18142 ts_tablet_manager.cc:1888] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 19:56:55.424727 18142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b61e94fc7e541c8aa84957560508a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.426723 17386 raft_consensus.cc:1050] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: attempting to promote NON_VOTER 2ca06dad25a645bca71b09741798194a to VOTER
I20240501 19:56:55.427376 17386 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 10, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:55.428705 14417 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a [term 2 LEARNER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:55.429088 17722 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:55.431203 12725 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:55.431639 17968 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:55.433482 17968 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 10 to 12, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.435729 14414 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
@ 0x7f897a6446ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
I20240501 19:56:55.438719 12444 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 2ca06dad25a645bca71b09741798194a reported cstate change: config changed from index 10 to 12, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
@ 0x7f897a64603b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55de9239375c std::function<>::operator()() at ??:0
I20240501 19:56:55.442835 12725 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.453600 18142 tablet_copy_client.cc:783] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:55.453639 12855 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:55.453830 18142 tablet_copy_client.cc:647] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:55.454272 17386 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
@ 0x7f897a633807 kudu::Thread::SuperviseThread() at ??:0
I20240501 19:56:55.455420 18142 tablet_copy_client.cc:523] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:55.458038 18142 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:55.458894 12855 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, 2ca06dad25a645bca71b09741798194a (127.5.139.70) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
@ 0x7f897ae716db start_thread at ??:0
I20240501 19:56:55.462317 12857 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
@ 0x7f89783a971f clone at ??:0
I20240501 19:56:55.463552 17322 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:55.468891 18128 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:55.469221 18128 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: Bootstrap complete.
I20240501 19:56:55.469404 18128 ts_tablet_manager.cc:1367] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: Time spent bootstrapping tablet: real 0.074s user 0.023s sys 0.007s
I20240501 19:56:55.469801 18128 raft_consensus.cc:348] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:55.469907 18128 raft_consensus.cc:729] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Initialized, Role: LEARNER
I20240501 19:56:55.469995 18128 consensus_queue.cc:260] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:55.470921 18128 ts_tablet_manager.cc:1398] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.473174 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session c6d2c7096e404ea9a6f73b123d74c622-557994658e2c4c8bbc1eeacdab958ddd received from {username='slave'} at 127.5.139.72:37741
I20240501 19:56:55.473248 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session c6d2c7096e404ea9a6f73b123d74c622-557994658e2c4c8bbc1eeacdab958ddd on tablet 557994658e2c4c8bbc1eeacdab958ddd with peer c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:55.496264 12592 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:55.497840 14286 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:55.498178 12725 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:55.498587 14548 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:55.498998 15568 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.499529 15654 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.499814 15654 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.502890 17140 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:55.505259 12725 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:55.505931 12445 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 8 to 10. New cstate: current_term: 2 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:55.510468 14548 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:55.512881 14286 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:55.519169 12592 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 19:56:55.521741 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): Couldn't send request to peer 2d52575d9e1b40dca0fc553d7295eb74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e4d79f5e11b4923acb2ce543884b54a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.521795 12725 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:55.523083 15569 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:55.523454 14548 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:55.523856 15569 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:55.526762 14286 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:55.527281 17140 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:55.527706 12725 consensus_queue.cc:237] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:55.527812 17140 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.529212 12592 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 19:56:55.529565 15926 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 19:56:55.530445 12662 consensus_peers.cc:485] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): Couldn't send request to peer 2d52575d9e1b40dca0fc553d7295eb74. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.531172 15472 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.531253 14548 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.531569 12595 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.535086 12449 catalog_manager.cc:5484] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 6 to 7, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:55.535223 12445 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 10 to 11, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New cstate: current_term: 2 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.537299 12439 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7e4d79f5e11b4923acb2ce543884b54a with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 19:56:55.538105 12592 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3"
dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49066
I20240501 19:56:55.538200 12592 raft_consensus.cc:595] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:55.538162 12725 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.538897 18142 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:55.539222 18142 tablet_bootstrap.cc:492] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap complete.
I20240501 19:56:55.539388 18142 ts_tablet_manager.cc:1367] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.081s user 0.032s sys 0.001s
I20240501 19:56:55.540081 18142 raft_consensus.cc:348] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:55.540205 18142 raft_consensus.cc:729] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: LEARNER
I20240501 19:56:55.540311 18142 consensus_queue.cc:260] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:55.540588 18142 ts_tablet_manager.cc:1398] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240501 19:56:55.540902 14286 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.549139 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-8b61e94fc7e541c8aa84957560508a31 received from {username='slave'} at 127.5.139.68:58755
I20240501 19:56:55.549216 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-8b61e94fc7e541c8aa84957560508a31 on tablet 8b61e94fc7e541c8aa84957560508a31 with peer 2d52575d9e1b40dca0fc553d7295eb74
I20240501 19:56:55.549494 12592 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:55.551995 14286 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 19:56:55.552191 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): Couldn't send request to peer 2d52575d9e1b40dca0fc553d7295eb74. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e4d79f5e11b4923acb2ce543884b54a. This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.552441 17140 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:55.552490 14548 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:55.553232 17140 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:55.555991 14548 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.561297 15569 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.561857 12451 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 11 to 12, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 2 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.565975 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e4d79f5e11b4923acb2ce543884b54a with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:55.566943 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4d79f5e11b4923acb2ce543884b54a with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:55.567243 18234 tablet_replica.cc:330] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:55.567378 18234 raft_consensus.cc:2229] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.567273 14286 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 19:56:55.567516 18234 raft_consensus.cc:2258] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.567734 17867 ts_tablet_manager.cc:902] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 19:56:55.568169 17867 tablet_copy_client.cc:308] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 19:56:55.568274 18234 ts_tablet_manager.cc:1875] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:55.569489 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 745e39ec23db4f90bb8b1e8b49a50244 from peer 58f750e4d9574c9ebfd3d9e57eefd411 ({username='slave'} at 127.5.139.66:43757)
I20240501 19:56:55.569602 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 745e39ec23db4f90bb8b1e8b49a50244 from peer 58f750e4d9574c9ebfd3d9e57eefd411 at {username='slave'} at 127.5.139.66:43757: session id = 58f750e4d9574c9ebfd3d9e57eefd411-745e39ec23db4f90bb8b1e8b49a50244
I20240501 19:56:55.570539 13007 tablet_copy_source_session.cc:205] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:55.570647 18234 ts_tablet_manager.cc:1888] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20240501 19:56:55.570703 18234 log.cc:1196] T 7e4d79f5e11b4923acb2ce543884b54a P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/7e4d79f5e11b4923acb2ce543884b54a
I20240501 19:56:55.571288 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet 7e4d79f5e11b4923acb2ce543884b54a (table rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]) successfully deleted
I20240501 19:56:55.571563 17867 ts_tablet_manager.cc:1875] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:56:55.573714 17867 ts_tablet_manager.cc:1888] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 19:56:55.574144 17867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 745e39ec23db4f90bb8b1e8b49a50244. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.575482 17867 tablet_copy_client.cc:783] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:55.575623 17867 tablet_copy_client.cc:647] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:55.579430 14286 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:56:55.579792 17322 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:55.582872 17867 tablet_copy_client.cc:523] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:55.584293 17867 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 19:56:55.586921 12986 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:56:55.589707 15472 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:55.594048 12725 consensus_queue.cc:237] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:55.595695 12857 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:55.596158 15926 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:55.597561 14286 raft_consensus.cc:1261] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:55.598394 15472 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.600420 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c7ddb6d79d384d39a709b5824f936d53 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:55.600881 12444 catalog_manager.cc:5484] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 7 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 2 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:55.601150 15926 consensus_queue.cc:1035] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:55.603014 12857 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.638948 18212 raft_consensus.cc:979] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: : Instructing follower 1d49e80be3d04fb99b711c62d61bac09 to start an election
I20240501 19:56:55.667891 17867 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:55.603504 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet c7ddb6d79d384d39a709b5824f936d53 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:55.687974 17867 tablet_bootstrap.cc:492] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap complete.
I20240501 19:56:55.688185 18347 tablet_replica.cc:330] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:55.688192 17867 ts_tablet_manager.cc:1367] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.104s user 0.026s sys 0.004s
I20240501 19:56:55.688318 18347 raft_consensus.cc:2229] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.688444 18347 raft_consensus.cc:2258] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.688688 17867 raft_consensus.cc:348] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 19:56:55.688834 17867 raft_consensus.cc:729] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: LEARNER
I20240501 19:56:55.688954 17867 consensus_queue.cc:260] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 19:56:55.689234 18347 ts_tablet_manager.cc:1875] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:55.689267 17867 ts_tablet_manager.cc:1398] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 19:56:55.689736 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-745e39ec23db4f90bb8b1e8b49a50244 received from {username='slave'} at 127.5.139.66:43757
I20240501 19:56:55.700652 12725 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ca3285abe3654723a6922672391e9423"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33106
I20240501 19:56:55.721760 12725 raft_consensus.cc:595] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:55.689800 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-745e39ec23db4f90bb8b1e8b49a50244 on tablet 745e39ec23db4f90bb8b1e8b49a50244 with peer 58f750e4d9574c9ebfd3d9e57eefd411
I20240501 19:56:55.691740 18347 ts_tablet_manager.cc:1888] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 19:56:55.722119 18347 log.cc:1196] T c7ddb6d79d384d39a709b5824f936d53 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/c7ddb6d79d384d39a709b5824f936d53
I20240501 19:56:55.723048 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet c7ddb6d79d384d39a709b5824f936d53 (table rebalance_test_table_7 [id=df4f757cd3d94896aaecbb3733403900]) successfully deleted
I20240501 19:56:55.709205 18142 ts_tablet_manager.cc:902] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:55.725765 18142 tablet_copy_client.cc:308] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:55.727600 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 8bc9fe9ade8246779164d6db31b11e16 from peer 2d52575d9e1b40dca0fc553d7295eb74 ({username='slave'} at 127.5.139.68:58755)
I20240501 19:56:55.727720 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 8bc9fe9ade8246779164d6db31b11e16 from peer 2d52575d9e1b40dca0fc553d7295eb74 at {username='slave'} at 127.5.139.68:58755: session id = 2d52575d9e1b40dca0fc553d7295eb74-8bc9fe9ade8246779164d6db31b11e16
I20240501 19:56:55.728657 12746 tablet_copy_source_session.cc:205] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:55.729987 18142 ts_tablet_manager.cc:1875] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:56:55.732163 18142 ts_tablet_manager.cc:1888] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 19:56:55.732367 18142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9fe9ade8246779164d6db31b11e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.733395 18142 tablet_copy_client.cc:783] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:55.733573 18142 tablet_copy_client.cc:647] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:55.734861 18142 tablet_copy_client.cc:523] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:55.736001 18142 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:55.717216 12857 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "04ccc8addc55441b9ab9d8265b90b03b"
dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42178
I20240501 19:56:55.753024 12857 raft_consensus.cc:595] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:55.605574 14286 raft_consensus.cc:2932] T c7ddb6d79d384d39a709b5824f936d53 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.639058 18212 raft_consensus.cc:1067] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Signalling peer 1d49e80be3d04fb99b711c62d61bac09 to start an election
I20240501 19:56:55.715533 17140 raft_consensus.cc:1050] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER 9069ec6fcdb248479037f2d37e0e851d to VOTER
I20240501 19:56:55.754534 17140 consensus_queue.cc:237] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
I20240501 19:56:55.648211 14285 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:55.756901 14547 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3"
dest_uuid: "1d49e80be3d04fb99b711c62d61bac09"
from {username='slave'} at 127.5.139.65:44979
I20240501 19:56:55.757088 14547 raft_consensus.cc:482] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:55.757155 14547 raft_consensus.cc:3037] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.758234 14547 raft_consensus.cc:504] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:55.759608 14547 leader_election.cc:288] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [CANDIDATE]: Term 2 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
I20240501 19:56:55.759981 12725 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:55.761648 12592 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "1d49e80be3d04fb99b711c62d61bac09" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415"
I20240501 19:56:55.761775 12592 raft_consensus.cc:3032] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:55.761817 12592 raft_consensus.cc:729] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Running, Role: LEADER
I20240501 19:56:55.761981 12592 consensus_queue.cc:260] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:55.762216 12592 raft_consensus.cc:3037] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.763409 12592 raft_consensus.cc:2454] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d49e80be3d04fb99b711c62d61bac09 in term 2.
I20240501 19:56:55.764631 14285 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:55.765000 12857 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240501 19:56:55.765386 18212 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:55.765662 14679 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:55.765866 17722 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 19:56:55.765906 12986 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "1d49e80be3d04fb99b711c62d61bac09" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:55.766023 12986 raft_consensus.cc:3037] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.767035 12986 raft_consensus.cc:2454] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d49e80be3d04fb99b711c62d61bac09 in term 2.
I20240501 19:56:55.767560 15569 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:55.767705 18212 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:55.768572 14485 leader_election.cc:302] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d49e80be3d04fb99b711c62d61bac09, 2d52575d9e1b40dca0fc553d7295eb74, ebbbd9ac07a94e8dbbae6def0a2df415; no voters:
I20240501 19:56:55.768847 17912 raft_consensus.cc:2781] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:55.768986 17912 raft_consensus.cc:686] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 LEADER]: Becoming Leader. State: Replica: 1d49e80be3d04fb99b711c62d61bac09, State: Running, Role: LEADER
I20240501 19:56:55.769256 18142 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:55.769241 17912 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:55.769527 18142 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap complete.
I20240501 19:56:55.769691 18142 ts_tablet_manager.cc:1367] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.034s user 0.024s sys 0.003s
I20240501 19:56:55.770129 18142 raft_consensus.cc:348] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:55.770298 18142 raft_consensus.cc:729] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: LEARNER
I20240501 19:56:55.770741 18142 consensus_queue.cc:260] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:55.771704 18142 ts_tablet_manager.cc:1398] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 19:56:55.772807 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-8bc9fe9ade8246779164d6db31b11e16 received from {username='slave'} at 127.5.139.68:58755
I20240501 19:56:55.772891 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-8bc9fe9ade8246779164d6db31b11e16 on tablet 8bc9fe9ade8246779164d6db31b11e16 with peer 2d52575d9e1b40dca0fc553d7295eb74
I20240501 19:56:55.773623 17722 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.773743 12444 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: term changed from 1 to 2, leader changed from ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) to 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71). New cstate: current_term: 2 leader_uuid: "1d49e80be3d04fb99b711c62d61bac09" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:55.774833 15569 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.776261 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf6948589837482a9f580731e3b861b8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:55.776953 14285 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.777254 12725 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.777868 12857 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.780126 14680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "1d49e80be3d04fb99b711c62d61bac09" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: true dest_uuid: "c6d2c7096e404ea9a6f73b123d74c622"
I20240501 19:56:55.780249 14680 raft_consensus.cc:3037] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.781023 12451 catalog_manager.cc:5484] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.781953 12445 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 8 to 10, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:55.785748 14680 raft_consensus.cc:2454] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d49e80be3d04fb99b711c62d61bac09 in term 2.
I20240501 19:56:55.786515 12574 tablet_service.cc:1514] Processing DeleteTablet for tablet cf6948589837482a9f580731e3b861b8 with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:55.787360 18416 tablet_replica.cc:330] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:55.787504 18416 raft_consensus.cc:2229] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.787617 18416 raft_consensus.cc:2258] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.788182 18416 ts_tablet_manager.cc:1875] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 19:56:55.789139 14221 connection.cc:535] client connection to 127.5.139.70:33933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:55.789268 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:55.789696 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.139.70:33933 (error 108)
W20240501 19:56:55.790218 12792 connection.cc:535] client connection to 127.5.139.70:33933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:55.790318 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:55.790412 12923 connection.cc:535] client connection to 127.5.139.70:33933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:55.790534 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:55.790675 12661 connection.cc:535] client connection to 127.5.139.70:33933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 19:56:55.790786 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:56:55.790828 18416 ts_tablet_manager.cc:1888] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
W20240501 19:56:55.791836 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:55.791916 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.792104 18416 log.cc:1196] T cf6948589837482a9f580731e3b861b8 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/cf6948589837482a9f580731e3b861b8
I20240501 19:56:55.792423 14679 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
W20240501 19:56:55.793464 12661 consensus_peers.cc:485] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:55.793529 12661 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.793991 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet cf6948589837482a9f580731e3b861b8 (table rebalance_test_table_8 [id=e77f8c1c9f524b539586b4509dd26f73]) successfully deleted
I20240501 19:56:55.794759 12592 consensus_queue.cc:237] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }
W20240501 19:56:55.794991 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:55.795567 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 19:56:55.795634 12792 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.796660 14285 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:55.797024 14679 raft_consensus.cc:1261] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:55.797463 15654 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 19:56:55.797576 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:55.797719 15568 consensus_queue.cc:1035] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:55.799278 15568 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.799592 14679 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.800063 14285 raft_consensus.cc:2932] T db240ec5f8b2444fb6114ad814f3cdb7 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } }
I20240501 19:56:55.800828 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db240ec5f8b2444fb6114ad814f3cdb7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:55.802673 12449 catalog_manager.cc:5484] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 8 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:55.804702 18095 raft_consensus.cc:979] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: : Instructing follower c6d2c7096e404ea9a6f73b123d74c622 to start an election
I20240501 19:56:55.804793 18095 raft_consensus.cc:1067] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Signalling peer c6d2c7096e404ea9a6f73b123d74c622 to start an election
I20240501 19:56:55.805584 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet db240ec5f8b2444fb6114ad814f3cdb7 with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:55.805784 18234 tablet_replica.cc:330] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:55.805935 18234 raft_consensus.cc:2229] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:55.806146 18234 raft_consensus.cc:2258] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:55.806978 14679 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "04ccc8addc55441b9ab9d8265b90b03b"
dest_uuid: "c6d2c7096e404ea9a6f73b123d74c622"
from {username='slave'} at 127.5.139.67:58385
I20240501 19:56:55.807122 18234 ts_tablet_manager.cc:1875] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:55.807145 14679 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:55.807207 14679 raft_consensus.cc:3037] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.807705 12725 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ca3285abe3654723a6922672391e9423"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33118
I20240501 19:56:55.807804 12725 raft_consensus.cc:595] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:55.808152 14679 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:55.809460 14679 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 2 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933)
I20240501 19:56:55.809980 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ccc8addc55441b9ab9d8265b90b03b" candidate_uuid: "c6d2c7096e404ea9a6f73b123d74c622" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:55.810101 12857 raft_consensus.cc:3032] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:55.810140 12857 raft_consensus.cc:729] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Running, Role: LEADER
I20240501 19:56:55.810151 18234 ts_tablet_manager.cc:1888] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:55.810292 12857 consensus_queue.cc:260] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:55.810541 12857 raft_consensus.cc:3037] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.811264 12857 raft_consensus.cc:2454] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d2c7096e404ea9a6f73b123d74c622 in term 2.
W20240501 19:56:55.812419 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:56:55.814361 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:56:55.815415 14615 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:55.815951 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ccc8addc55441b9ab9d8265b90b03b" candidate_uuid: "c6d2c7096e404ea9a6f73b123d74c622" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "9069ec6fcdb248479037f2d37e0e851d"
I20240501 19:56:55.816071 14285 raft_consensus.cc:3037] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.816936 14285 raft_consensus.cc:2454] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d2c7096e404ea9a6f73b123d74c622 in term 2.
I20240501 19:56:55.810215 18234 log.cc:1196] T db240ec5f8b2444fb6114ad814f3cdb7 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/db240ec5f8b2444fb6114ad814f3cdb7
I20240501 19:56:55.817270 14613 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 9069ec6fcdb248479037f2d37e0e851d, a238fc0c6d844cedb8adccb97d01ae5e, c6d2c7096e404ea9a6f73b123d74c622; no voters: 2ca06dad25a645bca71b09741798194a
I20240501 19:56:55.817545 16001 raft_consensus.cc:2781] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:55.817692 16001 raft_consensus.cc:686] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 2 LEADER]: Becoming Leader. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Running, Role: LEADER
I20240501 19:56:55.817759 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet db240ec5f8b2444fb6114ad814f3cdb7 (table rebalance_test_table_0 [id=f9f43040d1d7463b810225e4a7bce24c]) successfully deleted
I20240501 19:56:55.817927 16001 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:55.820564 12451 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: term changed from 1 to 2, leader changed from a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) to c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72). New cstate: current_term: 2 leader_uuid: "c6d2c7096e404ea9a6f73b123d74c622" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:55.828117 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.911901 18142 ts_tablet_manager.cc:902] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 19:56:55.912439 18142 tablet_copy_client.cc:308] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 19:56:55.912923 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet ae2885f7933f462c96db9a99a4eb2090 from peer 2d52575d9e1b40dca0fc553d7295eb74 ({username='slave'} at 127.5.139.68:58755)
I20240501 19:56:55.913026 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet ae2885f7933f462c96db9a99a4eb2090 from peer 2d52575d9e1b40dca0fc553d7295eb74 at {username='slave'} at 127.5.139.68:58755: session id = 2d52575d9e1b40dca0fc553d7295eb74-ae2885f7933f462c96db9a99a4eb2090
I20240501 19:56:55.913762 12746 tablet_copy_source_session.cc:205] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:55.914393 18142 ts_tablet_manager.cc:1875] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 19:56:55.916494 18142 ts_tablet_manager.cc:1888] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 19:56:55.916666 18142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae2885f7933f462c96db9a99a4eb2090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:55.917708 18142 tablet_copy_client.cc:783] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:55.917846 18142 tablet_copy_client.cc:647] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:55.918836 18142 tablet_copy_client.cc:523] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:55.919790 18142 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:55.923943 14679 raft_consensus.cc:1203] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240501 19:56:55.925254 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.942265 18142 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:55.942541 18142 tablet_bootstrap.cc:492] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap complete.
I20240501 19:56:55.942701 18142 ts_tablet_manager.cc:1367] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.023s user 0.021s sys 0.000s
I20240501 19:56:55.943115 18142 raft_consensus.cc:348] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:55.943240 18142 raft_consensus.cc:729] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: LEARNER
I20240501 19:56:55.943354 18142 consensus_queue.cc:260] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:55.943625 18142 ts_tablet_manager.cc:1398] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 19:56:55.944095 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-ae2885f7933f462c96db9a99a4eb2090 received from {username='slave'} at 127.5.139.68:58755
I20240501 19:56:55.944182 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-ae2885f7933f462c96db9a99a4eb2090 on tablet ae2885f7933f462c96db9a99a4eb2090 with peer 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:55.955489 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:55.960708 12986 raft_consensus.cc:1203] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20240501 19:56:55.995223 15926 raft_consensus.cc:979] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: : Instructing follower 2d52575d9e1b40dca0fc553d7295eb74 to start an election
I20240501 19:56:55.995326 15926 raft_consensus.cc:1067] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Signalling peer 2d52575d9e1b40dca0fc553d7295eb74 to start an election
I20240501 19:56:55.995770 12986 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ca3285abe3654723a6922672391e9423"
dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
from {username='slave'} at 127.5.139.66:43757
I20240501 19:56:55.995914 12986 raft_consensus.cc:482] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:55.995966 12986 raft_consensus.cc:3037] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.997102 12986 raft_consensus.cc:504] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:55.997542 12986 leader_election.cc:288] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 2 election: Requested vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 19:56:55.998108 12855 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3285abe3654723a6922672391e9423" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e"
I20240501 19:56:55.998212 12855 raft_consensus.cc:3037] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.998615 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3285abe3654723a6922672391e9423" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "1d49e80be3d04fb99b711c62d61bac09"
I20240501 19:56:55.998726 14547 raft_consensus.cc:3037] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:55.999256 12855 raft_consensus.cc:2454] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 2.
I20240501 19:56:55.999421 14547 raft_consensus.cc:2454] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 2.
I20240501 19:56:55.999801 18212 raft_consensus.cc:1050] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 to VOTER
I20240501 19:56:56.000042 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca3285abe3654723a6922672391e9423" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:56.000155 12725 raft_consensus.cc:3032] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:56.000195 12725 raft_consensus.cc:729] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:56.000195 18212 consensus_queue.cc:237] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 19:56:56.000331 12725 consensus_queue.cc:260] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:56.000511 12725 raft_consensus.cc:3037] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:56.001406 14679 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:56.001767 12725 raft_consensus.cc:2454] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 2.
I20240501 19:56:56.002219 12725 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:56.002581 15654 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:56.002686 14286 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } }
I20240501 19:56:56.002919 15654 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:56.004305 12855 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:56.004534 12921 leader_election.cc:302] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d49e80be3d04fb99b711c62d61bac09, 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:56.004755 18516 raft_consensus.cc:2781] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:56.004841 18516 raft_consensus.cc:686] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Becoming Leader. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:56.004987 18516 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:56.005494 12855 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:56.006040 15568 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:56.006093 17968 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:56.007073 12445 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: term changed from 1 to 2, leader changed from 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) to 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68). New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 19:56:56.009655 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:56.008930 17967 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:56.010583 15568 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.010879 12855 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.010946 14679 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.012748 12855 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } }
I20240501 19:56:56.013643 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e84a3865ce64eaebad4df46ec01842d with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:56.013768 12725 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.014218 12449 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 12 to 13, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:56.014315 12451 catalog_manager.cc:5484] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 7 to 8, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.017274 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e84a3865ce64eaebad4df46ec01842d with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:56.017460 18234 tablet_replica.cc:330] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:56.017589 18234 raft_consensus.cc:2229] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.017717 18234 raft_consensus.cc:2258] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.018464 18234 ts_tablet_manager.cc:1875] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:56.020773 12592 consensus_queue.cc:237] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 19:56:56.022102 14679 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:56.022375 12723 raft_consensus.cc:1261] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:56.022435 18212 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:56.022910 18212 consensus_queue.cc:1035] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:56.024114 18234 ts_tablet_manager.cc:1888] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 19:56:56.024247 18234 log.cc:1196] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/3e84a3865ce64eaebad4df46ec01842d
I20240501 19:56:56.024789 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet 3e84a3865ce64eaebad4df46ec01842d (table rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]) successfully deleted
I20240501 19:56:56.025104 18212 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.027026 12723 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.027549 14679 raft_consensus.cc:2932] T 544caa38514643d5b81cd6ed85e0bae7 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.027693 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 544caa38514643d5b81cd6ed85e0bae7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:56.028394 12444 catalog_manager.cc:5484] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 8 to 9, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:56.031610 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 544caa38514643d5b81cd6ed85e0bae7 with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:56.032800 18570 tablet_replica.cc:330] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:56.033020 18570 raft_consensus.cc:2229] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.033320 18570 raft_consensus.cc:2258] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.033936 18570 ts_tablet_manager.cc:1875] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:56.036127 18570 ts_tablet_manager.cc:1888] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 19:56:56.036257 18570 log.cc:1196] T 544caa38514643d5b81cd6ed85e0bae7 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/544caa38514643d5b81cd6ed85e0bae7
I20240501 19:56:56.036789 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 544caa38514643d5b81cd6ed85e0bae7 (table rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]) successfully deleted
I20240501 19:56:56.109673 18142 ts_tablet_manager.cc:902] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 19:56:56.110091 18142 tablet_copy_client.cc:308] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 19:56:56.110543 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet 7e4d79f5e11b4923acb2ce543884b54a from peer 2d52575d9e1b40dca0fc553d7295eb74 ({username='slave'} at 127.5.139.68:57561)
I20240501 19:56:56.110646 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet 7e4d79f5e11b4923acb2ce543884b54a from peer 2d52575d9e1b40dca0fc553d7295eb74 at {username='slave'} at 127.5.139.68:57561: session id = 2d52575d9e1b40dca0fc553d7295eb74-7e4d79f5e11b4923acb2ce543884b54a
I20240501 19:56:56.111343 12614 tablet_copy_source_session.cc:205] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 19:56:56.112147 18142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e4d79f5e11b4923acb2ce543884b54a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 19:56:56.114342 18142 tablet_copy_client.cc:783] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 0 data blocks...
I20240501 19:56:56.114495 18142 tablet_copy_client.cc:647] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 1 WAL segments...
I20240501 19:56:56.115619 18142 tablet_copy_client.cc:523] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 19:56:56.116706 18142 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 19:56:56.136189 18142 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 19:56:56.136461 18142 tablet_bootstrap.cc:492] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap complete.
I20240501 19:56:56.136637 18142 ts_tablet_manager.cc:1367] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.020s user 0.017s sys 0.000s
I20240501 19:56:56.137084 18142 raft_consensus.cc:348] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:56.137194 18142 raft_consensus.cc:729] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: LEARNER
I20240501 19:56:56.137300 18142 consensus_queue.cc:260] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 19:56:56.137550 18142 ts_tablet_manager.cc:1398] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 19:56:56.138111 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-7e4d79f5e11b4923acb2ce543884b54a received from {username='slave'} at 127.5.139.68:57561
I20240501 19:56:56.138202 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-7e4d79f5e11b4923acb2ce543884b54a on tablet 7e4d79f5e11b4923acb2ce543884b54a with peer 2d52575d9e1b40dca0fc553d7295eb74
I20240501 19:56:56.172832 12723 raft_consensus.cc:1203] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 19:56:56.224895 12986 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Refusing update from remote peer 1d49e80be3d04fb99b711c62d61bac09: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:56.225437 18409 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:56.255796 12592 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Refusing update from remote peer 1d49e80be3d04fb99b711c62d61bac09: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:56.256243 18409 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:56.258831 14679 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Refusing update from remote peer 1d49e80be3d04fb99b711c62d61bac09: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 19:56:56.259936 18673 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:56.270509 12986 raft_consensus.cc:1203] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240501 19:56:56.320106 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:56.337817 12986 raft_consensus.cc:1203] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 19:56:56.368248 15926 raft_consensus.cc:1050] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 to VOTER
I20240501 19:56:56.368639 15926 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 19:56:56.370028 12855 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 19:56:56.370154 14547 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 19:56:56.370504 12986 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 19:56:56.370540 15926 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 19:56:56.370764 15472 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 19:56:56.370976 17322 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 19:56:56.372737 15472 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.373185 14547 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.374562 12986 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.375218 14285 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:56.375648 18445 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:56.376757 12855 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:56.377130 12451 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 reported cstate change: config changed from index 15 to 16, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.379901 18445 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:56.380254 12723 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 1.16, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 19:56:56.381527 14547 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240501 19:56:56.381922 17322 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 19:56:56.382561 12855 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.16: config changed from index 15 to 16, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.383937 14547 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.384461 15472 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.385829 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8b61e94fc7e541c8aa84957560508a31 with cas_config_opid_index 16: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:56.386950 12445 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 16 to 17, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 19:56:56.388414 12986 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 16. Preceding OpId from leader: term: 1 index: 17. (index mismatch)
I20240501 19:56:56.388844 17322 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 19:56:56.389777 12986 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.17: config changed from index 16 to 17, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.390267 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b61e94fc7e541c8aa84957560508a31 with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:51566
I20240501 19:56:56.390453 18570 tablet_replica.cc:330] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 19:56:56.390585 18570 raft_consensus.cc:2229] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.390789 18570 raft_consensus.cc:2258] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.391731 18570 ts_tablet_manager.cc:1875] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:56.393723 18570 ts_tablet_manager.cc:1888] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.16
I20240501 19:56:56.393808 18570 log.cc:1196] T 8b61e94fc7e541c8aa84957560508a31 P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/8b61e94fc7e541c8aa84957560508a31
I20240501 19:56:56.394321 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 8b61e94fc7e541c8aa84957560508a31 (table rebalance_test_table_2 [id=2b92d6f20cdd41a4ac93f9cda49833c7]) successfully deleted
I20240501 19:56:56.398433 12857 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:56:56.398766 17877 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 19:56:56.418915 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:56.478266 17014 raft_consensus.cc:1050] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 to VOTER
I20240501 19:56:56.478686 17014 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:56.480177 14679 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:56.480309 14285 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:56.480831 17014 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 19:56:56.480953 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:56.481134 18741 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:56.483346 18748 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:56.483924 14285 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:56.486001 14679 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 19:56:56.486848 12451 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 10 to 11, c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:56.491274 12723 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:56:56.491955 17877 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:56.495169 14547 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 19:56:56.495608 18758 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:56.612268 15508 raft_consensus.cc:1050] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 to VOTER
I20240501 19:56:56.612660 15508 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 19:56:56.613929 12723 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:56.613958 14679 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:56.614003 14285 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:56.614354 18516 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:56.614537 15508 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:56.614737 17877 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:56.616784 17877 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.617067 12986 raft_consensus.cc:1203] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20240501 19:56:56.617452 14679 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.618182 14285 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.620003 12723 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.621460 12451 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 10 to 11, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:56.671327 13712 raft_consensus.cc:1050] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 to VOTER
I20240501 19:56:56.671737 13712 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 19:56:56.672914 12986 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:56.673023 14547 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 19:56:56.673384 15472 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 19:56:56.673537 13712 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 19:56:56.673734 12661 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:56.675542 15472 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.675953 14547 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.677690 12986 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.678112 12449 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 10 to 11, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:56.708676 14547 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:56.709164 18673 raft_consensus.cc:482] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2ca06dad25a645bca71b09741798194a)
I20240501 19:56:56.709278 18673 raft_consensus.cc:504] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:56.709714 18673 leader_election.cc:288] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
I20240501 19:56:56.709966 14679 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Refusing update from remote peer 1d49e80be3d04fb99b711c62d61bac09: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:56.709995 12986 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Refusing update from remote peer 1d49e80be3d04fb99b711c62d61bac09: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
W20240501 19:56:56.710255 14484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:56.710371 18673 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:56.710541 14679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7" candidate_uuid: "1d49e80be3d04fb99b711c62d61bac09" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "c6d2c7096e404ea9a6f73b123d74c622" is_pre_election: true
I20240501 19:56:56.710716 18673 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 19:56:56.711200 14484 leader_election.cc:334] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:56.711266 14484 leader_election.cc:302] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1d49e80be3d04fb99b711c62d61bac09; no voters: 2ca06dad25a645bca71b09741798194a, c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:56.711381 18409 raft_consensus.cc:2726] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 19:56:56.712359 17912 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 10 to 12, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:56.712788 12986 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:56.712872 14679 raft_consensus.cc:2932] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 10 to 12, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:56.714438 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 955edb7c12fa4b32ac58827f131931e3 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:56.715466 12444 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 10 to 12, VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) evicted. New cstate: current_term: 2 leader_uuid: "1d49e80be3d04fb99b711c62d61bac09" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:56.717677 12574 tablet_service.cc:1514] Processing DeleteTablet for tablet 955edb7c12fa4b32ac58827f131931e3 with delete_type TABLET_DATA_TOMBSTONED (TS ebbbd9ac07a94e8dbbae6def0a2df415 not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:43160
I20240501 19:56:56.718052 18832 tablet_replica.cc:330] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: stopping tablet replica
I20240501 19:56:56.718333 18832 raft_consensus.cc:2229] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:56.718660 18832 raft_consensus.cc:2258] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:56.719488 18832 ts_tablet_manager.cc:1875] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:56.721678 18832 ts_tablet_manager.cc:1888] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20240501 19:56:56.721760 18832 log.cc:1196] T 955edb7c12fa4b32ac58827f131931e3 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-0/wal/wals/955edb7c12fa4b32ac58827f131931e3
I20240501 19:56:56.722322 12439 catalog_manager.cc:4830] TS ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): tablet 955edb7c12fa4b32ac58827f131931e3 (table rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]) successfully deleted
I20240501 19:56:56.752406 18727 raft_consensus.cc:482] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 2ca06dad25a645bca71b09741798194a)
I20240501 19:56:56.752547 18727 raft_consensus.cc:504] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:56.753013 18727 leader_election.cc:288] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933), 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
W20240501 19:56:56.754411 14615 leader_election.cc:334] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:56.756879 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7" candidate_uuid: "c6d2c7096e404ea9a6f73b123d74c622" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "1d49e80be3d04fb99b711c62d61bac09" is_pre_election: true
I20240501 19:56:56.757089 14547 raft_consensus.cc:2454] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6d2c7096e404ea9a6f73b123d74c622 in term 2.
I20240501 19:56:56.757376 14616 leader_election.cc:302] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d49e80be3d04fb99b711c62d61bac09, c6d2c7096e404ea9a6f73b123d74c622; no voters: 2ca06dad25a645bca71b09741798194a
I20240501 19:56:56.757572 18727 raft_consensus.cc:2781] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 19:56:56.757637 18727 raft_consensus.cc:482] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 2ca06dad25a645bca71b09741798194a)
I20240501 19:56:56.757692 18727 raft_consensus.cc:3037] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Advancing to term 3
I20240501 19:56:56.758502 18727 raft_consensus.cc:504] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:56.758807 18727 leader_election.cc:288] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 3 election: Requested vote from peers 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933), 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 19:56:56.759160 14547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b15761177f4cffb7aa5e4d16e42ac7" candidate_uuid: "c6d2c7096e404ea9a6f73b123d74c622" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "1d49e80be3d04fb99b711c62d61bac09"
I20240501 19:56:56.759305 14547 raft_consensus.cc:3037] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Advancing to term 3
W20240501 19:56:56.759939 14615 leader_election.cc:334] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:56.760321 14547 raft_consensus.cc:2454] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d2c7096e404ea9a6f73b123d74c622 in term 3.
I20240501 19:56:56.760910 14616 leader_election.cc:302] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1d49e80be3d04fb99b711c62d61bac09, c6d2c7096e404ea9a6f73b123d74c622; no voters: 2ca06dad25a645bca71b09741798194a
I20240501 19:56:56.761087 18727 raft_consensus.cc:2781] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 19:56:56.761180 18727 raft_consensus.cc:686] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [term 3 LEADER]: Becoming Leader. State: Replica: c6d2c7096e404ea9a6f73b123d74c622, State: Running, Role: LEADER
I20240501 19:56:56.761322 18727 consensus_queue.cc:237] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 19:56:56.763490 12723 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bc9fe9ade8246779164d6db31b11e16"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33304
I20240501 19:56:56.763628 12723 raft_consensus.cc:595] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:56.763752 12445 catalog_manager.cc:5484] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: term changed from 2 to 3, leader changed from 2ca06dad25a645bca71b09741798194a (127.5.139.70) to c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72). New cstate: current_term: 3 leader_uuid: "c6d2c7096e404ea9a6f73b123d74c622" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:56.774566 12986 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244"
dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55098
I20240501 19:56:56.774698 12986 raft_consensus.cc:595] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:56.963304 13712 raft_consensus.cc:1050] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 to VOTER
I20240501 19:56:56.963658 13712 consensus_queue.cc:237] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 19:56:56.964869 12592 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:56.964891 12986 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 19:56:56.965359 17322 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 19:56:56.965490 15926 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 19:56:56.966111 12661 consensus_peers.cc:485] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:56.967239 15926 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.967650 12986 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.970279 12449 catalog_manager.cc:5484] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 7 to 8, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:56.971187 12986 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 19:56:56.971180 12592 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.972288 14547 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:56.972656 18516 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:56.972806 12857 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 19:56:56.973796 18516 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:56.974877 12723 consensus_queue.cc:237] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 19:56:56.976107 12592 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:56.975970 18516 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:56.977510 15472 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 19:56:56.977553 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca3285abe3654723a6922672391e9423 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:56.978170 12857 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:56.979998 12445 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 7 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:56.981321 15926 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.981810 12592 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.981812 14547 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } }
I20240501 19:56:56.983558 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae2885f7933f462c96db9a99a4eb2090 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:56.990157 12451 catalog_manager.cc:5484] T ae2885f7933f462c96db9a99a4eb2090 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.990969 12723 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8bc9fe9ade8246779164d6db31b11e16"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33324
I20240501 19:56:56.991094 12723 raft_consensus.cc:595] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Received request to transfer leadership
I20240501 19:56:56.991406 12704 tablet_service.cc:1514] Processing DeleteTablet for tablet ca3285abe3654723a6922672391e9423 with delete_type TABLET_DATA_TOMBSTONED (TS 58f750e4d9574c9ebfd3d9e57eefd411 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:49730
I20240501 19:56:56.993636 12986 raft_consensus.cc:1261] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 19:56:56.994213 13712 consensus_queue.cc:1035] T ae2885f7933f462c96db9a99a4eb2090 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 19:56:56.994333 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:56:56.995491 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:56.996667 12986 raft_consensus.cc:2932] T ae2885f7933f462c96db9a99a4eb2090 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:56.997634 15508 raft_consensus.cc:979] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: : Instructing follower 58f750e4d9574c9ebfd3d9e57eefd411 to start an election
I20240501 19:56:56.997715 15508 raft_consensus.cc:1067] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Signalling peer 58f750e4d9574c9ebfd3d9e57eefd411 to start an election
I20240501 19:56:56.998100 12723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244"
dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
from {username='slave'} at 127.5.139.68:58755
I20240501 19:56:56.998209 12723 raft_consensus.cc:482] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:56.998276 12723 raft_consensus.cc:3037] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:56.999202 12723 raft_consensus.cc:504] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 19:56:56.999644 12723 leader_election.cc:288] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 2 election: Requested vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
I20240501 19:56:57.000123 14285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "9069ec6fcdb248479037f2d37e0e851d"
I20240501 19:56:57.000249 14285 raft_consensus.cc:3037] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:57.000285 14679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "c6d2c7096e404ea9a6f73b123d74c622"
I20240501 19:56:57.000407 14679 raft_consensus.cc:3037] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:57.001122 14285 raft_consensus.cc:2454] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 2.
I20240501 19:56:57.001226 14679 raft_consensus.cc:2454] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 2.
I20240501 19:56:57.001655 12661 leader_election.cc:302] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622; no voters:
I20240501 19:56:57.001803 13712 raft_consensus.cc:2781] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:57.001883 13712 raft_consensus.cc:686] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEADER]: Becoming Leader. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:57.002012 13712 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 19:56:57.003198 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 19:56:57.003335 12988 raft_consensus.cc:3032] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:57.003388 12988 raft_consensus.cc:729] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:57.003542 12988 consensus_queue.cc:260] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 19:56:57.003772 12988 raft_consensus.cc:3037] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:57.004616 12988 raft_consensus.cc:2454] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58f750e4d9574c9ebfd3d9e57eefd411 in term 2.
I20240501 19:56:57.006167 12444 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: term changed from 1 to 2, leader changed from 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) to 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66). New cstate: current_term: 2 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:57.008713 12988 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "745e39ec23db4f90bb8b1e8b49a50244"
dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55112
I20240501 19:56:57.008832 12988 raft_consensus.cc:595] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Received request to transfer leadership
I20240501 19:56:57.008891 12988 raft_consensus.cc:603] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20240501 19:56:57.009232 18918 tablet_replica.cc:330] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: stopping tablet replica
I20240501 19:56:57.009384 18918 raft_consensus.cc:2229] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:57.010200 18918 raft_consensus.cc:2258] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.010712 18212 raft_consensus.cc:1050] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 to VOTER
I20240501 19:56:57.011121 18918 ts_tablet_manager.cc:1875] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:57.011130 18212 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 19:56:57.012581 14285 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:57.012704 14547 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:57.013023 15569 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:57.013322 15568 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:57.013736 18918 ts_tablet_manager.cc:1888] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 19:56:57.013794 18918 log.cc:1196] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-1/wal/wals/ca3285abe3654723a6922672391e9423
I20240501 19:56:57.014271 12437 catalog_manager.cc:4830] TS 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): tablet ca3285abe3654723a6922672391e9423 (table rebalance_test_table_5 [id=109bfb23537140259f6e685179d80ba0]) successfully deleted
I20240501 19:56:57.014561 12988 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:57.014976 15568 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:57.016121 15569 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:57.017235 12988 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:57.017720 14285 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:57.020066 12444 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 12 to 13, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:57.020145 14547 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
W20240501 19:56:57.022862 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:57.024576 12592 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 19:56:57.025745 12988 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 19:56:57.026072 18212 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:57.026366 14547 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 19:56:57.026835 18212 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:57.027997 18937 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:57.028465 12988 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:57.028623 14548 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 19:56:57.030803 12449 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 13 to 14, VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New cstate: current_term: 2 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:57.031035 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7e4d79f5e11b4923acb2ce543884b54a with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:57.033535 14266 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e4d79f5e11b4923acb2ce543884b54a with delete_type TABLET_DATA_TOMBSTONED (TS 9069ec6fcdb248479037f2d37e0e851d not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:54698
I20240501 19:56:57.033898 18941 tablet_replica.cc:330] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: stopping tablet replica
I20240501 19:56:57.034307 18941 raft_consensus.cc:2229] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:57.034541 18941 raft_consensus.cc:2258] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:57.035485 18941 ts_tablet_manager.cc:1875] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:57.037348 18941 ts_tablet_manager.cc:1888] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20240501 19:56:57.037420 18941 log.cc:1196] T 7e4d79f5e11b4923acb2ce543884b54a P 9069ec6fcdb248479037f2d37e0e851d: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/wals/7e4d79f5e11b4923acb2ce543884b54a
I20240501 19:56:57.037899 12437 catalog_manager.cc:4830] TS 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): tablet 7e4d79f5e11b4923acb2ce543884b54a (table rebalance_test_table_1 [id=6f10e15b627248ecb203396d1403dcdb]) successfully deleted
W20240501 19:56:57.145819 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:57.146623 14548 raft_consensus.cc:1261] T e4b15761177f4cffb7aa5e4d16e42ac7 P 1d49e80be3d04fb99b711c62d61bac09 [term 3 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 19:56:57.146987 18725 consensus_queue.cc:1035] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 19:56:57.184772 13712 raft_consensus.cc:979] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411: : Instructing follower 2d52575d9e1b40dca0fc553d7295eb74 to start an election
I20240501 19:56:57.184878 13712 raft_consensus.cc:1067] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Signalling peer 2d52575d9e1b40dca0fc553d7295eb74 to start an election
I20240501 19:56:57.185330 12988 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8bc9fe9ade8246779164d6db31b11e16"
dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
from {username='slave'} at 127.5.139.66:43757
I20240501 19:56:57.185477 12988 raft_consensus.cc:482] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 19:56:57.185577 12988 raft_consensus.cc:3037] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:57.186705 12988 raft_consensus.cc:504] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 19:56:57.187155 12988 leader_election.cc:288] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 2 election: Requested vote from peers 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895), 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933), 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 19:56:57.187603 12723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9fe9ade8246779164d6db31b11e16" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 19:56:57.187706 12723 raft_consensus.cc:3032] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 19:56:57.187742 12723 raft_consensus.cc:729] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Running, Role: LEADER
I20240501 19:56:57.187695 14548 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bc9fe9ade8246779164d6db31b11e16" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 2 candidate_status { last_received { term: 1 index: 11 } } ignore_live_leader: true dest_uuid: "1d49e80be3d04fb99b711c62d61bac09"
I20240501 19:56:57.187820 14548 raft_consensus.cc:3037] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:57.187899 12723 consensus_queue.cc:260] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 19:56:57.188040 12723 raft_consensus.cc:3037] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Advancing to term 2
I20240501 19:56:57.188637 14548 raft_consensus.cc:2454] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 2.
I20240501 19:56:57.188793 12723 raft_consensus.cc:2454] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 2.
I20240501 19:56:57.189241 12924 leader_election.cc:302] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 1d49e80be3d04fb99b711c62d61bac09, 2d52575d9e1b40dca0fc553d7295eb74, 58f750e4d9574c9ebfd3d9e57eefd411; no voters:
I20240501 19:56:57.189428 17877 raft_consensus.cc:2781] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 19:56:57.189517 17877 raft_consensus.cc:686] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Becoming Leader. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 19:56:57.189656 17877 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
W20240501 19:56:57.192036 12923 leader_election.cc:334] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:57.192409 12448 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: term changed from 1 to 2, leader changed from 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) to 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68). New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:57.236197 12704 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 19:56:57.236449 14529 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 19:56:57.237448 14660 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:57.238668 12574 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 19:56:57.245357 12967 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:57.248983 12836 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 19:56:57.259239 14266 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 19:56:57.302497 14731 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskuLQw07/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 19:56:48.214253 14734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:48.214468 14734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:48.217916 14734 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:49.229295 14734 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 19:56:49.697542 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.697657 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.697700 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.697773 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.697813 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.697861 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.697896 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.697952 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.697983 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.698012 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.698047 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.698076 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.698107 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.698150 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:49.731163 14734 rebalancer_tool.cc:1074] tablet 851eeb0ab3364529907d9d09f2a4022e: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:49.753832 14734 rebalancer_tool.cc:1074] tablet e4b15761177f4cffb7aa5e4d16e42ac7: '58f750e4d9574c9ebfd3d9e57eefd411' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:49.795812 14734 rebalancer_tool.cc:1074] tablet 0fb481d394034a8f85af613beb5c9ec8: '2d52575d9e1b40dca0fc553d7295eb74' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:49.840356 14734 rebalancer_tool.cc:1074] tablet 706878734216445f8422e422228d4230: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:49.847930 14734 rebalancer_tool.cc:1084] tablet f08535cb2f46464b9e510a45fee67b4f: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '1d49e80be3d04fb99b711c62d61bac09' move ignored: Illegal state: tablet f08535cb2f46464b9e510a45fee67b4f: replica 1d49e80be3d04fb99b711c62d61bac09 is already present
I20240501 19:56:49.848021 14734 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:50.351958 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.352075 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.352152 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.352248 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.352355 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.352442 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.352505 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.352555 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.352758 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.352842 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.352900 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:50.437052 14734 rebalancer_tool.cc:1074] tablet 8bc9fe9ade8246779164d6db31b11e16: '2d52575d9e1b40dca0fc553d7295eb74' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:50.547467 14734 rebalancer_tool.cc:1074] tablet f08535cb2f46464b9e510a45fee67b4f: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:50.692911 14734 rebalancer_tool.cc:1084] tablet 32a17c2e392a49b7b181fa7a25a778a1: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move ignored: Illegal state: tablet 32a17c2e392a49b7b181fa7a25a778a1: replica 1d49e80be3d04fb99b711c62d61bac09 is already present
I20240501 19:56:50.692992 14734 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:51.583000 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:51.583163 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server 2ca06dad25a645bca71b09741798194a to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:51.583225 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:51.583359 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:51.583439 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:51.583564 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:51.583678 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:51.583741 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:51.583791 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:51.583843 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:51.583894 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:51.583954 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:51.584026 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:51.584095 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:51.663511 14734 rebalancer_tool.cc:1084] tablet 22b272a8582d4060a6a527ac4fce9ae5: '58f750e4d9574c9ebfd3d9e57eefd411' -> '1d49e80be3d04fb99b711c62d61bac09' move ignored: Not found: Peer with uuid 58f750e4d9574c9ebfd3d9e57eefd411 not found in consensus config
I20240501 19:56:51.663619 14734 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:52.357015 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:52.357142 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:52.357198 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:52.357234 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:52.357313 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:52.357350 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:52.357383 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:52.357415 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:52.357458 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:52.383675 14734 rebalancer_tool.cc:1074] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move scheduled
I20240501 19:56:52.419755 14734 rebalancer_tool.cc:1074] tablet 800d5f43cd1b4fb6bd8bbcac262aeb80: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:52.451330 14734 rebalancer_tool.cc:1074] tablet 04ccc8addc55441b9ab9d8265b90b03b: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:52.490475 14734 rebalancer_tool.cc:1074] tablet 7d7ea564ea504163a1e17561be9a5130: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move scheduled
I20240501 19:56:52.533429 14734 rebalancer_tool.cc:1074] tablet 8b61e94fc7e541c8aa84957560508a31: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:52.567710 14734 rebalancer_tool.cc:1074] tablet c7ddb6d79d384d39a709b5824f936d53: '2d52575d9e1b40dca0fc553d7295eb74' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:52.664250 14734 rebalancer_tool.cc:1074] tablet 61bc22cdc6ab45529066fe6f4d14d4c7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:52.930673 14734 rebalancer_tool.cc:1084] tablet 557994658e2c4c8bbc1eeacdab958ddd: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '9069ec6fcdb248479037f2d37e0e851d' move ignored: Illegal state: tablet 557994658e2c4c8bbc1eeacdab958ddd: replica 9069ec6fcdb248479037f2d37e0e851d is already present
I20240501 19:56:52.930768 14734 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:53.704952 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:53.705121 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server 9069ec6fcdb248479037f2d37e0e851d to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:53.705161 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:53.718950 14734 rebalancer_tool.cc:1084] tablet 955edb7c12fa4b32ac58827f131931e3: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '1d49e80be3d04fb99b711c62d61bac09' move ignored: Illegal state: tablet 955edb7c12fa4b32ac58827f131931e3: replica 1d49e80be3d04fb99b711c62d61bac09 is already present
I20240501 19:56:53.719043 14734 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:54.348421 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:54.348610 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:54.348677 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
I20240501 19:56:54.361013 14734 rebalancer_tool.cc:1084] tablet 7e4d79f5e11b4923acb2ce543884b54a: '9069ec6fcdb248479037f2d37e0e851d' -> '' move ignored: Remote error: Service unavailable: leader transfer in progress
I20240501 19:56:54.361124 14734 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:54.776690 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server 9069ec6fcdb248479037f2d37e0e851d to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:54.776788 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 9069ec6fcdb248479037f2d37e0e851d to server 2d52575d9e1b40dca0fc553d7295eb74
I20240501 19:56:54.830760 14734 rebalancer_tool.cc:1084] tablet 22b272a8582d4060a6a527ac4fce9ae5: '2d52575d9e1b40dca0fc553d7295eb74' -> '2ca06dad25a645bca71b09741798194a' move ignored: Illegal state: tablet 22b272a8582d4060a6a527ac4fce9ae5: replica 2ca06dad25a645bca71b09741798194a is already present
I20240501 19:56:54.830833 14734 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:55.447479 14734 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2d52575d9e1b40dca0fc553d7295eb74
I20240501 19:56:55.505461 14734 rebalancer_tool.cc:1074] tablet 7e4d79f5e11b4923acb2ce543884b54a: '9069ec6fcdb248479037f2d37e0e851d' -> '' move scheduled
I20240501 19:56:55.534070 14734 rebalancer_tool.cc:1074] tablet ae2885f7933f462c96db9a99a4eb2090: '2ca06dad25a645bca71b09741798194a' -> '2d52575d9e1b40dca0fc553d7295eb74' move scheduled
I20240501 19:56:55.591501 14734 rebalancer_tool.cc:1136] tablet 61bc22cdc6ab45529066fe6f4d14d4c7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '2ca06dad25a645bca71b09741798194a' move completed: Incomplete: tablet 61bc22cdc6ab45529066fe6f4d14d4c7, TS a238fc0c6d844cedb8adccb97d01ae5e -> TS 2ca06dad25a645bca71b09741798194a move failed, target replica disappeared
I20240501 19:56:55.661126 14734 rebalancer_tool.cc:1136] tablet c7ddb6d79d384d39a709b5824f936d53: '2d52575d9e1b40dca0fc553d7295eb74' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:55.679503 14734 rebalancer_tool.cc:1136] tablet 7d7ea564ea504163a1e17561be9a5130: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move completed: OK
I20240501 19:56:55.695299 14734 rebalancer_tool.cc:1136] tablet 8b61e94fc7e541c8aa84957560508a31: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move completed: Incomplete: tablet 8b61e94fc7e541c8aa84957560508a31, TS ebbbd9ac07a94e8dbbae6def0a2df415 -> TS 2ca06dad25a645bca71b09741798194a move failed, target replica disappeared
I20240501 19:56:55.709435 14734 rebalancer_tool.cc:1136] tablet 04ccc8addc55441b9ab9d8265b90b03b: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
W20240501 19:56:55.791880 18398 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 19:56:55.739961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.139.70:33933 (local address 127.0.0.1:52164)
0501 19:56:55.745277 (+ 5316us) negotiation.cc:107] Waiting for socket to connect
0501 19:56:55.745294 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 19:56:55.745409 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 19:56:55.791022 (+ 45613us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":5229,"thread_start_us":82,"threads_started":1}
I20240501 19:56:55.792865 14734 rebalancer_tool.cc:1123] tablet e4b15761177f4cffb7aa5e4d16e42ac7: '58f750e4d9574c9ebfd3d9e57eefd411' -> '2ca06dad25a645bca71b09741798194a' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 19:56:55.808898 14734 rebalancer_tool.cc:1136] tablet 851eeb0ab3364529907d9d09f2a4022e: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 19:56:55.822525 14734 rebalancer_tool.cc:1136] tablet 0fb481d394034a8f85af613beb5c9ec8: '2d52575d9e1b40dca0fc553d7295eb74' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
I20240501 19:56:55.834270 14734 rebalancer_tool.cc:1136] tablet 800d5f43cd1b4fb6bd8bbcac262aeb80: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 19:56:55.840507 14734 rebalancer_tool.cc:1136] tablet 706878734216445f8422e422228d4230: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:55.845642 14734 rebalancer_tool.cc:1123] tablet 8bc9fe9ade8246779164d6db31b11e16: '2d52575d9e1b40dca0fc553d7295eb74' -> '2ca06dad25a645bca71b09741798194a' move is abandoned: Illegal state: 8bc9fe9ade8246779164d6db31b11e16: source replica 2d52575d9e1b40dca0fc553d7295eb74 does not have REPLACE attribute set
I20240501 19:56:55.850589 14734 rebalancer_tool.cc:1136] tablet f08535cb2f46464b9e510a45fee67b4f: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:56.077821 14734 rebalancer_tool.cc:1136] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move completed: OK
I20240501 19:56:57.150333 14734 rebalancer_tool.cc:1136] tablet 7e4d79f5e11b4923acb2ce543884b54a: '9069ec6fcdb248479037f2d37e0e851d' -> '' move completed: OK
I20240501 19:56:57.161005 14734 rebalancer_tool.cc:1136] tablet ae2885f7933f462c96db9a99a4eb2090: '2ca06dad25a645bca71b09741798194a' -> '2d52575d9e1b40dca0fc553d7295eb74' move completed: OK
Illegal state: tablet server 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): unacceptable health status UNAVAILABLE
I20240501 19:56:57.483163 14679 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:57.483582 17322 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:57.528205 12988 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:57.528632 17322 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:57.531801 14285 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:57.532441 15472 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 19:56:57.566449 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:56:57.577245 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 19:56:57.598018 14548 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 1d49e80be3d04fb99b711c62d61bac09 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:57.598425 17877 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 19:56:57.808080 12723 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 19:56:57.808504 17877 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 19:56:57.989457 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:58.024785 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:56:58.063771 12723 consensus_queue.cc:237] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 19:56:58.065081 14679 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:58.065172 14285 raft_consensus.cc:1261] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 19:56:58.065474 19132 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:58.065708 19133 consensus_queue.cc:1035] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 19:56:58.067190 15472 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:58.067636 14679 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:58.067687 14285 raft_consensus.cc:2932] T 745e39ec23db4f90bb8b1e8b49a50244 P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 19:56:58.069698 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 745e39ec23db4f90bb8b1e8b49a50244 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 19:56:58.070412 12447 catalog_manager.cc:5484] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 11 to 13, VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) evicted. New cstate: current_term: 2 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 19:56:58.072917 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 745e39ec23db4f90bb8b1e8b49a50244 with delete_type TABLET_DATA_TOMBSTONED (TS 2d52575d9e1b40dca0fc553d7295eb74 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:49210
I20240501 19:56:58.073261 19137 tablet_replica.cc:330] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: stopping tablet replica
I20240501 19:56:58.073421 19137 raft_consensus.cc:2229] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 19:56:58.073550 19137 raft_consensus.cc:2258] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 19:56:58.074115 19137 ts_tablet_manager.cc:1875] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 19:56:58.075874 19137 ts_tablet_manager.cc:1888] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20240501 19:56:58.075942 19137 log.cc:1196] T 745e39ec23db4f90bb8b1e8b49a50244 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-3/wal/wals/745e39ec23db4f90bb8b1e8b49a50244
I20240501 19:56:58.076462 12440 catalog_manager.cc:4830] TS 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): tablet 745e39ec23db4f90bb8b1e8b49a50244 (table rebalance_test_table_4 [id=c5dde3ea07b74da4b047a58bf0339e0d]) successfully deleted
W20240501 19:56:58.101451 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:58.139657 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:58.161271 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:58.205665 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:56:58.480279 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:58.606920 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:58.647157 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:56:58.894230 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:58.961154 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:59.235719 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:56:59.273553 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:56:59.513062 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:56:59.689375 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:00.138100 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 19:57:00.260746 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:00.442224 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:00.519600 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:00.826769 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:00.827487 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:00.832006 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:00.840426 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:57:00.841090 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:00.895414 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:00.981642 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 19:57:01.008495 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:01.085901 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:01.162623 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:01.261791 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 19:57:01.372660 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:01.420056 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:01.439517 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:01.763962 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:01.859901 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:02.001102 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:02.412312 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:02.473577 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 19:57:02.997009 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:03.118044 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:03.308975 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:03.358528 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:03.475697 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:03.578464 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:03.593315 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:03.597569 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:03.851905 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:04.097726 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:04.187816 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:04.510710 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:04.694722 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:04.906733 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 19:57:05.273910 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:05.315035 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:05.490620 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 19:57:05.491393 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:05.760175 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:05.792003 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:05.833237 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:05.852062 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:05.908978 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:05.909678 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:06.058311 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:06.058984 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:06.178901 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 19:57:06.181007 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:06.210873 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:57:06.296797 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:06.380077 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:06.533444 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:57:06.534185 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:06.943861 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:07.324301 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:07.397070 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:07.427421 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 19:57:07.797165 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:07.872984 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:08.066185 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:08.308881 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:08.384721 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:08.485975 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:08.492944 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:08.700886 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:08.840893 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:08.861734 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:08.970847 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:09.262259 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:09.542691 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:09.781291 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:10.037088 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 19:57:10.348397 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:10.367945 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:10.447309 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:10.760907 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:10.880805 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:11.092012 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:11.092700 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:11.142612 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:11.178314 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 19:57:11.278250 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:11.364868 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:11.424582 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:11.446125 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:11.530658 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:57:11.577070 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:11.590709 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 19:57:11.852128 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:57:11.930760 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:12.053818 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:12.492062 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 19:57:12.507541 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:12.736507 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:13.125926 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:13.184363 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:13.301590 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:13.541617 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:13.697115 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:13.697804 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:13.702900 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:13.717450 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:13.965713 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:13.987046 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:14.103440 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:14.177613 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:14.592453 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:14.625586 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:15.008709 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 19:57:15.038661 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:15.238240 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:15.391782 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:15.608278 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:15.722460 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:16.010455 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:16.097610 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 19:57:16.104856 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:16.472442 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:16.473089 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:16.483796 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:16.564615 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:16.598999 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:57:16.604050 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:16.644647 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:16.717447 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:16.769924 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:16.859643 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 19:57:17.150266 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:17.198486 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:17.262331 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 19:57:17.668406 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:17.671288 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:18.067287 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:18.114506 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:18.153949 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:18.433538 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:18.558588 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:18.905859 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:18.941140 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:18.943027 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:18.997738 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:19.349390 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:19.422937 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:19.474296 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:19.726594 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 19:57:19.962260 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:20.240880 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:20.378031 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:20.429780 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:20.736038 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:20.787978 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:20.814383 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:21.264186 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:57:21.264802 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:21.500697 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:21.534461 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:21.535245 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:21.564821 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:21.646584 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:21.670867 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:21.797392 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:21.977589 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:22.174877 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:22.192804 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 19:57:22.193421 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 19:57:22.335604 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:22.471210 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:22.647315 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:22.666409 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:22.845710 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:23.232312 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:23.287568 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:23.347880 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:23.765115 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:23.772707 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:23.842468 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:24.089480 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:24.252161 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:24.521023 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:24.668694 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:24.773128 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:24.899060 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 19:57:25.053687 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:25.062479 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:25.241618 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:25.670696 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:25.819447 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:25.999987 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:26.048465 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:26.132818 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:26.567986 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:26.568614 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:26.576455 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:26.577167 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:26.593708 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:26.842583 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:26.956542 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:26.960749 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:57:27.114499 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:27.381682 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:57:27.382541 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:27.443038 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 19:57:27.446848 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:27.480370 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:27.501925 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:27.509529 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:27.957531 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:28.384461 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:28.404675 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:28.486121 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:28.641844 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:28.758662 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:29.091154 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:29.290087 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:29.592381 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:29.642311 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:29.918994 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:30.026670 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:30.107654 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:30.110019 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:30.110689 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:30.132576 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 19:57:30.244361 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:30.627149 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:30.821269 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:30.829216 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:31.050505 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:31.327531 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:31.580116 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:31.580688 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:31.775712 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:31.800848 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:31.801425 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:31.993755 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:32.125880 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:32.155119 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:32.185609 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:32.390100 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:57:32.390684 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:32.394973 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:32.491114 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 19:57:32.632404 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:32.670150 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:32.685245 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:32.978688 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:33.076149 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:33.378334 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:33.614007 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:33.621881 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:34.130182 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:34.304381 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:34.362509 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:34.527593 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:34.757227 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:34.906745 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:34.925318 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:35.041819 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 19:57:35.136670 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:35.327327 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:35.511454 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:35.549710 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:35.897607 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:36.074522 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:36.138449 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:36.753576 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:36.754617 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:36.926707 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:37.033355 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:37.064569 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:37.065331 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:37.221895 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:37.258159 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:37.342828 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:37.390352 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:57:37.390978 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:37.469991 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:57:37.556125 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 19:57:37.681013 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:37.710294 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:37.872890 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:57:37.898363 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:38.062270 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:38.478379 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:38.619697 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:38.663727 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:39.071741 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:39.328642 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:39.363310 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:39.620898 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:39.676211 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:39.769984 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:39.783270 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:40.025833 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 19:57:40.066726 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:40.280807 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:40.411278 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:40.556289 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:40.789614 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:41.058354 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:41.200452 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:41.450652 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:41.707801 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:41.946229 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:41.946835 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:41.950227 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:41.958369 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:42.261013 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:42.317207 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:42.362607 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:42.443791 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:42.476336 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:57:42.579514 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:57:42.580055 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:42.615892 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:42.710338 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 19:57:42.957587 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:42.993227 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:43.180706 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:57:43.360307 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:43.555789 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:43.686216 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:44.078323 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:44.271596 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:44.455368 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:44.550129 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:44.550525 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:44.834347 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:44.906599 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:45.008659 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:45.029078 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:45.038035 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 19:57:45.303556 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:45.610517 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:45.964496 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:46.040956 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:46.175165 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:46.560572 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:46.933066 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:47.058758 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:47.082257 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:47.127748 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:47.157953 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:57:47.158643 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:47.170874 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:47.385444 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:47.386076 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:47.544639 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:57:47.589350 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:47.712157 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:47.804497 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 19:57:47.868237 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 19:57:47.981087 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 19:57:48.140384 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:48.315066 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:48.390535 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:48.409456 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:48.718041 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:49.219537 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:49.254877 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:49.365900 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:49.400292 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:49.536605 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:49.581369 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:49.706246 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:49.932986 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:49.938097 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:50.311491 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:50.394670 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 19:57:50.698014 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:50.704629 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:50.745599 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:51.158730 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:51.546896 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:51.597980 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:51.608103 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:52.030164 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:52.037181 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:52.069553 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:52.135658 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:52.248165 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:52.248865 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:52.464922 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:57:52.568020 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:57:52.618364 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:52.644125 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:52.981947 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 19:57:53.054009 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:53.084040 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:53.179405 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:53.246016 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:53.256975 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:53.512382 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:53.655187 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:54.026578 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:54.130697 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:54.331547 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:54.483319 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:54.526849 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:54.621665 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:54.652308 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:54.904031 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:55.040355 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:55.309429 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:55.440780 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:55.494668 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 19:57:55.685760 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:55.879161 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:56.006609 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:56.481309 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:56.555984 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:57:56.854076 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:56.891695 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:56.898818 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:56.976579 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:57.197482 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:57.281536 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:57:57.436220 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:57.436830 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:57.645296 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:57:57.687752 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:57:57.863989 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:57.914979 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:58.190692 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 19:57:58.200852 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:57:58.350092 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:57:58.384732 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:57:58.471740 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:58.527055 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:57:58.840422 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:57:58.872753 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:57:58.900188 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:59.258363 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:57:59.302651 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:57:59.403538 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:57:59.490734 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:57:59.571336 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:57:59.834751 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:57:59.955837 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:57:59.966399 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:00.527742 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:00.662389 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 19:58:01.046226 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:01.048134 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:01.224547 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:01.427078 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:01.446673 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:01.548633 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:01.748145 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:01.963615 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:02.091310 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:02.313397 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:02.338001 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:02.375213 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:02.378773 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:02.773588 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:02.774268 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:02.800653 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:58:02.919519 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:02.920122 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:02.947309 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:58:03.256933 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:03.261389 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 19:58:03.386922 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:03.502436 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:03.518968 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:03.720530 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:03.776037 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:58:03.884965 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:04.071719 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:58:04.072302 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:04.327126 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:04.442970 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:04.539765 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:04.650790 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:04.802282 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:04.888772 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:04.891116 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:05.355195 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:05.761618 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:05.836850 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:05.854490 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 19:58:06.049247 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:06.253398 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:06.508256 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:06.637260 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:06.773008 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:06.773150 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:07.057644 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:07.060214 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:07.066496 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:07.489856 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:07.567242 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:07.641148 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:07.817629 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:58:08.008332 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:08.008983 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:08.125119 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:08.216470 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:08.445585 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 19:58:08.597918 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:08.605764 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:08.640796 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:58:08.962476 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:58:08.975234 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:09.180212 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:09.295722 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:09.296448 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:09.364895 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:09.456650 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:09.552793 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:09.626484 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:10.034334 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:10.132526 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:10.138137 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:10.621551 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:10.770431 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:10.907289 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 19:58:10.992003 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:11.024144 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:11.145452 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:11.413473 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:11.794680 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:11.813953 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:11.827857 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:11.841209 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:11.954427 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:12.271057 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:12.435969 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:12.524976 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:12.976531 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:13.139876 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 19:58:13.203253 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 19:58:13.258713 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:13.259398 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:13.300034 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:13.368454 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:58:13.567487 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:13.586222 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:13.683370 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:14.073434 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:14.168627 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:58:14.169227 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:14.391510 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:14.423967 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:14.424654 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:14.455571 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:14.597797 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:14.636382 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:14.900828 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:14.980018 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:15.442893 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:15.777197 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 19:58:15.942054 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:15.952695 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:15.969283 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:15.992151 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:16.537068 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:16.590930 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:16.600939 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:16.883214 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:16.986830 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:17.155148 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:17.159689 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:17.272143 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:17.629405 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:18.084396 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:18.262897 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:58:18.263571 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 19:58:18.309168 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:18.467304 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:18.513528 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:58:18.514201 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:18.593716 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:18.594352 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:18.769927 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:19.015239 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:19.186188 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:19.187106 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:19.237380 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:19.428704 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:19.474133 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:19.536808 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:19.537504 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:19.760810 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:19.997126 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:20.050065 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:20.390480 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:20.639793 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:20.754151 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 19:58:20.798794 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:20.802538 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:21.201167 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:21.291643 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:21.682940 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:21.877068 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:21.992120 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:22.198760 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:22.212922 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:22.237242 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:22.239961 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:22.258791 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:22.670977 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:23.100776 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:23.170749 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:23.349850 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:23.492496 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:58:23.493219 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 19:58:23.599112 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:23.724856 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:58:23.725589 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:23.742964 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:23.866091 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:58:24.229597 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:24.361579 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:24.412405 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:24.413069 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:24.576463 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:24.666290 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:24.698463 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:24.843501 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:24.844230 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:24.944765 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:25.109388 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:25.614840 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:25.691630 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:25.807446 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:25.972923 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:26.127921 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 19:58:26.216889 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:26.619063 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:26.676203 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:26.860147 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:26.922848 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:26.925575 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:27.050796 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:27.352928 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:27.388312 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:27.664937 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:28.043570 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:28.253849 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:28.465945 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:28.506172 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:28.701817 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:58:28.705911 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 19:58:28.733379 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:28.827693 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:28.916438 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:58:28.933003 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:29.047919 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:29.069540 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:29.221782 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:29.518204 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:29.558079 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:29.677219 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:29.677953 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:29.878100 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:29.880918 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:29.881590 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:30.085008 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:30.629222 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:30.751605 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:30.938369 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:31.013635 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:31.146186 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:31.338814 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 19:58:31.435204 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:31.463225 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:31.725458 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:31.999177 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:32.152516 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:32.243064 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:32.425168 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:32.532984 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:32.535727 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:33.106978 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:33.129292 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:33.369922 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:33.673513 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:33.702824 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:58:33.703490 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:33.853483 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 19:58:33.948845 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:33.949494 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:34.131455 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:34.132056 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:34.242815 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:58:34.293972 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:34.376605 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:34.730350 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:34.823900 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:34.928166 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:34.928778 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:35.131124 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:35.131897 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:35.266117 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:35.738448 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:35.775530 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:35.799090 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:58:36.061403 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:36.147243 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:36.355966 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:36.364760 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:36.565173 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:36.763381 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 19:58:36.929764 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:58:36.984099 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:37.265142 12442 maintenance_manager.cc:419] P be1305e487d64b748456a721443e3b3b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033363
W20240501 19:58:37.283828 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 19:58:37.283818 12441 maintenance_manager.cc:643] P be1305e487d64b748456a721443e3b3b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.012s sys 0.005s Metrics: {"bytes_written":20242,"cfile_init":1,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":165,"dirs.run_wall_time_us":1038,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":488,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":136}
I20240501 19:58:37.284653 12442 maintenance_manager.cc:419] P be1305e487d64b748456a721443e3b3b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11860 bytes on disk
I20240501 19:58:37.285163 12441 maintenance_manager.cc:643] P be1305e487d64b748456a721443e3b3b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240501 19:58:37.350929 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:37.602691 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:37.906414 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:38.006173 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:38.200912 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:58:38.305439 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:58:38.550436 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:38.563326 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:58:38.726150 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:58:38.726830 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:58:38.954164 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:58:39.024468 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:39.048193 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 19:58:39.051559 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:39.193814 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:58:39.280052 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:58:39.469427 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:58:39.598064 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:39.940048 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:39.940871 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:58:39.965484 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:58:40.116497 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:40.200397 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 19:58:40.571137 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:58:40.601611 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:40.623003 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:58:40.790731 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:58:41.086160 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:41.128680 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:58:41.343026 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:58:41.433416 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 19:58:41.506918 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:58:41.610942 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:42.015516 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:58:42.208690 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:42.257938 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:58:42.447885 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(04a70a2ee2144f2287795ffe723d2716): perf score=0.033336
I20240501 19:58:42.472898 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(04a70a2ee2144f2287795ffe723d2716) complete. Timing: real 0.025s user 0.023s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":173,"dirs.queue_time_us":210,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":946,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":372,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":313,"threads_started":3,"wal-append.queue_time_us":229}
I20240501 19:58:42.474143 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(04a70a2ee2144f2287795ffe723d2716): 4834 bytes on disk
I20240501 19:58:42.474601 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(04a70a2ee2144f2287795ffe723d2716) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 19:58:42.515408 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(04a70a2ee2144f2287795ffe723d2716): perf score=0.033353
I20240501 19:58:42.549867 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(04a70a2ee2144f2287795ffe723d2716) complete. Timing: real 0.034s user 0.022s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":164,"dirs.queue_time_us":204,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1640,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":527,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":322,"threads_started":3,"wal-append.queue_time_us":194}
I20240501 19:58:42.551112 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(04a70a2ee2144f2287795ffe723d2716): 4834 bytes on disk
I20240501 19:58:42.551584 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(04a70a2ee2144f2287795ffe723d2716) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
I20240501 19:58:42.597349 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(12a50be843fb49ee8a66c9a17d271d61): perf score=0.033372
W20240501 19:58:42.634122 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:58:42.648290 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(12a50be843fb49ee8a66c9a17d271d61) complete. Timing: real 0.051s user 0.032s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":18435,"dirs.queue_time_us":11201,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1606,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":4207,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":280,"threads_started":3,"wal-append.queue_time_us":3829}
I20240501 19:58:42.649905 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(12a50be843fb49ee8a66c9a17d271d61): 4921 bytes on disk
I20240501 19:58:42.650557 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(12a50be843fb49ee8a66c9a17d271d61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240501 19:58:42.659911 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(db240ec5f8b2444fb6114ad814f3cdb7): perf score=0.033389
W20240501 19:58:42.660094 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 19:58:42.697762 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(db240ec5f8b2444fb6114ad814f3cdb7) complete. Timing: real 0.037s user 0.025s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":961,"dirs.queue_time_us":7798,"dirs.run_cpu_time_us":182,"dirs.run_wall_time_us":3281,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":327,"threads_started":3,"wal-append.queue_time_us":397}
I20240501 19:58:42.699331 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(db240ec5f8b2444fb6114ad814f3cdb7): 4581 bytes on disk
I20240501 19:58:42.700336 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(db240ec5f8b2444fb6114ad814f3cdb7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240501 19:58:42.812961 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:43.053905 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(3e84a3865ce64eaebad4df46ec01842d): perf score=0.033353
I20240501 19:58:43.080344 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(3e84a3865ce64eaebad4df46ec01842d) complete. Timing: real 0.026s user 0.023s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":395,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1351,"drs_written":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4,"lbm_write_time_us":772,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":142}
I20240501 19:58:43.081130 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(3e84a3865ce64eaebad4df46ec01842d): 4921 bytes on disk
I20240501 19:58:43.081449 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(3e84a3865ce64eaebad4df46ec01842d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240501 19:58:43.126442 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:58:43.182765 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:58:43.202200 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(7e4d79f5e11b4923acb2ce543884b54a): perf score=0.033394
I20240501 19:58:43.215296 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(7e4d79f5e11b4923acb2ce543884b54a) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1436,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":462,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":150,"threads_started":2,"wal-append.queue_time_us":129}
I20240501 19:58:43.215974 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(7e4d79f5e11b4923acb2ce543884b54a): 4834 bytes on disk
I20240501 19:58:43.216347 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(7e4d79f5e11b4923acb2ce543884b54a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 19:58:43.274844 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:43.403101 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(8b61e94fc7e541c8aa84957560508a31): perf score=0.033348
I20240501 19:58:43.415027 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(8b61e94fc7e541c8aa84957560508a31) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1609,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":444,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":134}
I20240501 19:58:43.415712 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(8b61e94fc7e541c8aa84957560508a31): 4921 bytes on disk
I20240501 19:58:43.416029 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(8b61e94fc7e541c8aa84957560508a31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 19:58:43.467484 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(7d7ea564ea504163a1e17561be9a5130): perf score=0.033363
I20240501 19:58:43.485085 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(7d7ea564ea504163a1e17561be9a5130) complete. Timing: real 0.017s user 0.013s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1479,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":648,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":93,"threads_started":1,"wal-append.queue_time_us":160}
I20240501 19:58:43.486074 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(7d7ea564ea504163a1e17561be9a5130): 4834 bytes on disk
I20240501 19:58:43.486512 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(7d7ea564ea504163a1e17561be9a5130) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
I20240501 19:58:43.583113 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(0fb481d394034a8f85af613beb5c9ec8): perf score=0.033398
I20240501 19:58:43.594981 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(0fb481d394034a8f85af613beb5c9ec8) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":169,"dirs.run_wall_time_us":1461,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":507,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":131,"threads_started":2,"wal-append.queue_time_us":126}
I20240501 19:58:43.595620 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(0fb481d394034a8f85af613beb5c9ec8): 4581 bytes on disk
I20240501 19:58:43.595986 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(0fb481d394034a8f85af613beb5c9ec8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240501 19:58:43.623250 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:58:43.801740 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:58:43.802397 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 19:58:43.863852 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:43.884348 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:58:43.916329 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:58:43.979420 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(706878734216445f8422e422228d4230): perf score=0.033334
I20240501 19:58:43.988392 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(32a17c2e392a49b7b181fa7a25a778a1): perf score=0.033337
I20240501 19:58:43.992199 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(706878734216445f8422e422228d4230) complete. Timing: real 0.013s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":2423,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":437,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":133}
I20240501 19:58:43.992858 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(706878734216445f8422e422228d4230): 4581 bytes on disk
I20240501 19:58:43.993223 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(706878734216445f8422e422228d4230) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 19:58:43.993916 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(d75bd8f836b947248ee124f456c7d721): perf score=0.033337
I20240501 19:58:44.002054 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(32a17c2e392a49b7b181fa7a25a778a1) complete. Timing: real 0.013s user 0.008s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":162,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1407,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":481,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":147,"threads_started":2,"wal-append.queue_time_us":128}
I20240501 19:58:44.002791 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(32a17c2e392a49b7b181fa7a25a778a1): 4921 bytes on disk
I20240501 19:58:44.003140 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(32a17c2e392a49b7b181fa7a25a778a1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:58:44.013999 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(d75bd8f836b947248ee124f456c7d721) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":160,"dirs.run_wall_time_us":1265,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":359,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":78,"threads_started":1,"wal-append.queue_time_us":162}
I20240501 19:58:44.014735 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(d75bd8f836b947248ee124f456c7d721): 4834 bytes on disk
I20240501 19:58:44.015069 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(d75bd8f836b947248ee124f456c7d721) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:58:44.097529 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(d75bd8f836b947248ee124f456c7d721): perf score=0.033366
I20240501 19:58:44.111537 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(d75bd8f836b947248ee124f456c7d721) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":124,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1567,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":524,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":108,"threads_started":2,"wal-append.queue_time_us":121}
I20240501 19:58:44.112445 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(d75bd8f836b947248ee124f456c7d721): 4834 bytes on disk
I20240501 19:58:44.112855 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(d75bd8f836b947248ee124f456c7d721) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240501 19:58:44.147670 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 19:58:44.168371 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(32a17c2e392a49b7b181fa7a25a778a1): perf score=0.033387
I20240501 19:58:44.180613 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(32a17c2e392a49b7b181fa7a25a778a1) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":173,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":488,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":119}
I20240501 19:58:44.181345 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(32a17c2e392a49b7b181fa7a25a778a1): 4921 bytes on disk
I20240501 19:58:44.181660 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(32a17c2e392a49b7b181fa7a25a778a1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 19:58:44.400938 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:44.484822 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:58:44.504765 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(544caa38514643d5b81cd6ed85e0bae7): perf score=0.033339
I20240501 19:58:44.515878 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(544caa38514643d5b81cd6ed85e0bae7) complete. Timing: real 0.011s user 0.008s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":209,"dirs.run_wall_time_us":1370,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":369,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":174}
I20240501 19:58:44.516552 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(544caa38514643d5b81cd6ed85e0bae7): 4581 bytes on disk
I20240501 19:58:44.516906 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(851eeb0ab3364529907d9d09f2a4022e): perf score=0.033346
I20240501 19:58:44.516928 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(544caa38514643d5b81cd6ed85e0bae7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 19:58:44.531823 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(851eeb0ab3364529907d9d09f2a4022e) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":141,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":2424,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":385,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":132,"threads_started":2,"wal-append.queue_time_us":126}
I20240501 19:58:44.532688 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(851eeb0ab3364529907d9d09f2a4022e): 4834 bytes on disk
I20240501 19:58:44.533033 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(851eeb0ab3364529907d9d09f2a4022e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
W20240501 19:58:44.720067 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:58:44.745676 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 19:58:44.933946 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(800d5f43cd1b4fb6bd8bbcac262aeb80): perf score=0.033346
W20240501 19:58:44.937527 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:44.938311 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:44.951164 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(800d5f43cd1b4fb6bd8bbcac262aeb80) complete. Timing: real 0.017s user 0.011s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1726,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1670,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":602,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":454}
I20240501 19:58:44.951880 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(800d5f43cd1b4fb6bd8bbcac262aeb80): 4920 bytes on disk
I20240501 19:58:44.952464 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(800d5f43cd1b4fb6bd8bbcac262aeb80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 19:58:45.018483 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(22b272a8582d4060a6a527ac4fce9ae5): perf score=0.033367
I20240501 19:58:45.030663 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(22b272a8582d4060a6a527ac4fce9ae5) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1437,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":398,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":142,"threads_started":2,"wal-append.queue_time_us":102}
I20240501 19:58:45.031373 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(22b272a8582d4060a6a527ac4fce9ae5): 4834 bytes on disk
I20240501 19:58:45.031728 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(22b272a8582d4060a6a527ac4fce9ae5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
I20240501 19:58:45.034726 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(22b272a8582d4060a6a527ac4fce9ae5): perf score=0.033371
I20240501 19:58:45.046967 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(22b272a8582d4060a6a527ac4fce9ae5) complete. Timing: real 0.012s user 0.008s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":159,"dirs.run_cpu_time_us":167,"dirs.run_wall_time_us":1331,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":435,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":132,"threads_started":2,"wal-append.queue_time_us":121}
I20240501 19:58:45.047611 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(ca3285abe3654723a6922672391e9423): perf score=0.033372
I20240501 19:58:45.060503 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(ca3285abe3654723a6922672391e9423) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":375,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":83,"threads_started":1,"wal-append.queue_time_us":397}
I20240501 19:58:45.061534 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(22b272a8582d4060a6a527ac4fce9ae5): 4834 bytes on disk
I20240501 19:58:45.061980 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(22b272a8582d4060a6a527ac4fce9ae5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 19:58:45.062666 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(ca3285abe3654723a6922672391e9423): 4581 bytes on disk
I20240501 19:58:45.062990 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(ca3285abe3654723a6922672391e9423) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
I20240501 19:58:45.115787 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(ca3285abe3654723a6922672391e9423): perf score=0.033391
I20240501 19:58:45.128170 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(ca3285abe3654723a6922672391e9423) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":145,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1477,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":477,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":125,"threads_started":2,"wal-append.queue_time_us":116}
I20240501 19:58:45.128898 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(ca3285abe3654723a6922672391e9423): 4581 bytes on disk
I20240501 19:58:45.129273 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(ca3285abe3654723a6922672391e9423) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
W20240501 19:58:45.149919 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:58:45.246649 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:58:45.307865 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:58:45.333928 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:45.435170 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:58:45.454530 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(61bc22cdc6ab45529066fe6f4d14d4c7): perf score=0.033354
I20240501 19:58:45.467033 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(61bc22cdc6ab45529066fe6f4d14d4c7) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":152,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1491,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":445,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":136}
I20240501 19:58:45.467789 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(61bc22cdc6ab45529066fe6f4d14d4c7): 4830 bytes on disk
I20240501 19:58:45.468276 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(61bc22cdc6ab45529066fe6f4d14d4c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240501 19:58:45.533586 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(61bc22cdc6ab45529066fe6f4d14d4c7): perf score=0.033376
I20240501 19:58:45.545482 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(61bc22cdc6ab45529066fe6f4d14d4c7) complete. Timing: real 0.012s user 0.010s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":150,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1453,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":389,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":126}
I20240501 19:58:45.546098 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(61bc22cdc6ab45529066fe6f4d14d4c7): 4830 bytes on disk
I20240501 19:58:45.546437 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(61bc22cdc6ab45529066fe6f4d14d4c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240501 19:58:45.565032 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(955edb7c12fa4b32ac58827f131931e3): perf score=0.033387
I20240501 19:58:45.582818 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(955edb7c12fa4b32ac58827f131931e3) complete. Timing: real 0.018s user 0.014s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":123,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1399,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":449,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":107,"threads_started":2,"wal-append.queue_time_us":110}
I20240501 19:58:45.583655 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(955edb7c12fa4b32ac58827f131931e3): 4581 bytes on disk
I20240501 19:58:45.585310 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(955edb7c12fa4b32ac58827f131931e3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 19:58:45.595448 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:58:45.596146 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 19:58:45.631006 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(557994658e2c4c8bbc1eeacdab958ddd): perf score=0.033401
I20240501 19:58:45.642768 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(557994658e2c4c8bbc1eeacdab958ddd) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":198,"dirs.run_cpu_time_us":203,"dirs.run_wall_time_us":1491,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":392,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":178,"threads_started":2,"wal-append.queue_time_us":206}
I20240501 19:58:45.643512 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(557994658e2c4c8bbc1eeacdab958ddd): 4921 bytes on disk
I20240501 19:58:45.643836 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(557994658e2c4c8bbc1eeacdab958ddd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4}
W20240501 19:58:45.712334 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:45.797559 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(ce6add2f19a649b3a12497951047705c): perf score=0.033338
I20240501 19:58:45.808543 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(ce6add2f19a649b3a12497951047705c) complete. Timing: real 0.011s user 0.008s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":53,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1520,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":374,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":65,"threads_started":1,"wal-append.queue_time_us":140}
I20240501 19:58:45.809279 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(ce6add2f19a649b3a12497951047705c): 4581 bytes on disk
I20240501 19:58:45.809598 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(ce6add2f19a649b3a12497951047705c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:58:45.894908 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(ce6add2f19a649b3a12497951047705c): perf score=0.033368
I20240501 19:58:45.906550 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(ce6add2f19a649b3a12497951047705c) complete. Timing: real 0.011s user 0.007s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1374,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":465,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":125,"threads_started":1,"wal-append.queue_time_us":195}
I20240501 19:58:45.907274 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(ce6add2f19a649b3a12497951047705c): 4581 bytes on disk
I20240501 19:58:45.907598 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(ce6add2f19a649b3a12497951047705c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 19:58:45.908130 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(c7ddb6d79d384d39a709b5824f936d53): perf score=0.033369
I20240501 19:58:45.922741 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(c7ddb6d79d384d39a709b5824f936d53) complete. Timing: real 0.014s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":135,"dirs.run_wall_time_us":1266,"drs_written":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4,"lbm_write_time_us":443,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":91,"threads_started":1,"wal-append.queue_time_us":152}
I20240501 19:58:45.923329 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(c7ddb6d79d384d39a709b5824f936d53): 4834 bytes on disk
I20240501 19:58:45.923652 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(c7ddb6d79d384d39a709b5824f936d53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240501 19:58:45.970228 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(8bc9fe9ade8246779164d6db31b11e16): perf score=0.033390
I20240501 19:58:45.982017 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(8bc9fe9ade8246779164d6db31b11e16) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":190,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1469,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":455,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":151,"threads_started":2,"wal-append.queue_time_us":120}
I20240501 19:58:45.982926 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(c7ddb6d79d384d39a709b5824f936d53): perf score=0.033392
I20240501 19:58:46.004935 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(c7ddb6d79d384d39a709b5824f936d53) complete. Timing: real 0.022s user 0.011s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1515,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":507,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":86,"threads_started":1,"wal-append.queue_time_us":153}
I20240501 19:58:46.005590 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(8bc9fe9ade8246779164d6db31b11e16): 4921 bytes on disk
I20240501 19:58:46.005892 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(8bc9fe9ade8246779164d6db31b11e16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 19:58:46.006604 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(c7ddb6d79d384d39a709b5824f936d53): 4834 bytes on disk
I20240501 19:58:46.006955 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(c7ddb6d79d384d39a709b5824f936d53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 19:58:46.068682 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:58:46.087146 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(f08535cb2f46464b9e510a45fee67b4f): perf score=0.033335
I20240501 19:58:46.098788 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(f08535cb2f46464b9e510a45fee67b4f) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":186,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1552,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":127}
I20240501 19:58:46.099453 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(f08535cb2f46464b9e510a45fee67b4f): 4922 bytes on disk
I20240501 19:58:46.099778 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(f08535cb2f46464b9e510a45fee67b4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240501 19:58:46.137578 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:46.174638 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(cf6948589837482a9f580731e3b861b8): perf score=0.033358
I20240501 19:58:46.186364 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(cf6948589837482a9f580731e3b861b8) complete. Timing: real 0.012s user 0.007s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1603,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":412,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":88,"threads_started":1,"wal-append.queue_time_us":154}
I20240501 19:58:46.187026 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(cf6948589837482a9f580731e3b861b8): 4834 bytes on disk
I20240501 19:58:46.187346 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(cf6948589837482a9f580731e3b861b8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240501 19:58:46.258220 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(f08535cb2f46464b9e510a45fee67b4f): perf score=0.033384
I20240501 19:58:46.270202 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(f08535cb2f46464b9e510a45fee67b4f) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":243,"dirs.run_wall_time_us":1403,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":404,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":99,"threads_started":1,"wal-append.queue_time_us":184}
I20240501 19:58:46.270952 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(f08535cb2f46464b9e510a45fee67b4f): 4922 bytes on disk
I20240501 19:58:46.271310 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(f08535cb2f46464b9e510a45fee67b4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 19:58:46.311187 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(d940b16d07d54945abe4a10a11dd6a2a): perf score=0.033399
I20240501 19:58:46.322161 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(d940b16d07d54945abe4a10a11dd6a2a) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":150,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1521,"drs_written":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4,"lbm_write_time_us":369,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":128,"threads_started":2,"wal-append.queue_time_us":184}
I20240501 19:58:46.322793 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(d940b16d07d54945abe4a10a11dd6a2a): 4582 bytes on disk
I20240501 19:58:46.323119 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(d940b16d07d54945abe4a10a11dd6a2a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240501 19:58:46.595918 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:58:46.629771 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:46.643750 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 19:58:46.712946 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 19:58:46.772881 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(ae2885f7933f462c96db9a99a4eb2090): perf score=0.033341
I20240501 19:58:46.784672 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(ae2885f7933f462c96db9a99a4eb2090) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":175,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1456,"drs_written":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4,"lbm_write_time_us":412,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":123,"threads_started":2,"wal-append.queue_time_us":124}
I20240501 19:58:46.785573 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(ae2885f7933f462c96db9a99a4eb2090): 4834 bytes on disk
I20240501 19:58:46.785959 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(ae2885f7933f462c96db9a99a4eb2090) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 19:58:46.829591 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling FlushMRSOp(ae2885f7933f462c96db9a99a4eb2090): perf score=0.033354
I20240501 19:58:46.846427 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: FlushMRSOp(ae2885f7933f462c96db9a99a4eb2090) complete. Timing: real 0.017s user 0.012s sys 0.002s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":141,"dirs.run_cpu_time_us":176,"dirs.run_wall_time_us":1442,"drs_written":1,"lbm_read_time_us":39,"lbm_reads_lt_1ms":4,"lbm_write_time_us":486,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":106,"threads_started":2,"wal-append.queue_time_us":117}
I20240501 19:58:46.847060 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(ae2885f7933f462c96db9a99a4eb2090): 4834 bytes on disk
I20240501 19:58:46.847412 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(ae2885f7933f462c96db9a99a4eb2090) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240501 19:58:46.855115 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:58:46.939504 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(04ccc8addc55441b9ab9d8265b90b03b): perf score=0.033389
I20240501 19:58:46.950379 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(04ccc8addc55441b9ab9d8265b90b03b) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":147,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1351,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":381,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":115}
I20240501 19:58:46.951064 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(04ccc8addc55441b9ab9d8265b90b03b): 4579 bytes on disk
I20240501 19:58:46.951391 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(04ccc8addc55441b9ab9d8265b90b03b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:58:46.951942 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling FlushMRSOp(0bf64d9f3ce94db28b032cdcd795a255): perf score=0.033392
I20240501 19:58:46.962978 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: FlushMRSOp(0bf64d9f3ce94db28b032cdcd795a255) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":63,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1235,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":425,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":768,"thread_start_us":59,"threads_started":1,"wal-append.queue_time_us":124}
I20240501 19:58:46.963532 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(0bf64d9f3ce94db28b032cdcd795a255): 4921 bytes on disk
I20240501 19:58:46.963853 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(0bf64d9f3ce94db28b032cdcd795a255) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 19:58:47.057840 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:47.274422 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:58:47.440485 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:47.586328 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:58:47.724967 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:58:47.872272 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:48.054340 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:58:48.251860 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:48.308140 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:58:48.619491 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:58:48.725430 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:49.133724 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:58:49.191813 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:58:49.223830 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:58:49.236066 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:58:49.307538 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:49.312474 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 19:58:49.402734 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:58:49.685840 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:49.843883 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:58:49.992729 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:50.000813 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 19:58:50.143028 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:58:50.279258 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:50.356809 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240501 19:58:50.544409 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(800d5f43cd1b4fb6bd8bbcac262aeb80): perf score=0.033397
W20240501 19:58:50.572365 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:58:50.576905 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(800d5f43cd1b4fb6bd8bbcac262aeb80) complete. Timing: real 0.032s user 0.026s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":160,"dirs.queue_time_us":260,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1072,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1187,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":208,"threads_started":3,"wal-append.queue_time_us":141}
I20240501 19:58:50.578222 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(800d5f43cd1b4fb6bd8bbcac262aeb80): 4920 bytes on disk
I20240501 19:58:50.578716 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(800d5f43cd1b4fb6bd8bbcac262aeb80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 19:58:50.579447 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(706878734216445f8422e422228d4230): perf score=0.033370
I20240501 19:58:50.592931 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(706878734216445f8422e422228d4230) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1435,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":385,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":110,"threads_started":1,"wal-append.queue_time_us":185}
I20240501 19:58:50.593799 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(706878734216445f8422e422228d4230): 4581 bytes on disk
I20240501 19:58:50.594144 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(706878734216445f8422e422228d4230) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240501 19:58:50.611213 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:58:50.620934 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(8b61e94fc7e541c8aa84957560508a31): perf score=0.033389
I20240501 19:58:50.655853 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(8b61e94fc7e541c8aa84957560508a31) complete. Timing: real 0.034s user 0.031s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":180,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":888,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":253,"threads_started":3,"wal-append.queue_time_us":158}
I20240501 19:58:50.657073 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(8b61e94fc7e541c8aa84957560508a31): 4921 bytes on disk
I20240501 19:58:50.657629 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(8b61e94fc7e541c8aa84957560508a31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
W20240501 19:58:50.742269 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:50.742995 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:58:50.796427 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(d940b16d07d54945abe4a10a11dd6a2a): perf score=0.033377
W20240501 19:58:50.813380 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:50.829015 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(d940b16d07d54945abe4a10a11dd6a2a) complete. Timing: real 0.032s user 0.030s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":180,"dirs.queue_time_us":1327,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":614,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":309,"threads_started":3,"wal-append.queue_time_us":213}
I20240501 19:58:50.830531 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(22b272a8582d4060a6a527ac4fce9ae5): perf score=0.033386
I20240501 19:58:50.845777 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(745e39ec23db4f90bb8b1e8b49a50244): perf score=0.033388
I20240501 19:58:50.849692 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(22b272a8582d4060a6a527ac4fce9ae5) complete. Timing: real 0.019s user 0.017s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":147,"dirs.run_wall_time_us":1315,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":522,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":115,"threads_started":1,"wal-append.queue_time_us":199}
I20240501 19:58:50.850744 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(0fb481d394034a8f85af613beb5c9ec8): perf score=0.033386
I20240501 19:58:50.866336 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(745e39ec23db4f90bb8b1e8b49a50244) complete. Timing: real 0.020s user 0.016s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":39,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1285,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":623,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":97,"threads_started":1,"wal-append.queue_time_us":174}
I20240501 19:58:50.867092 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(04ccc8addc55441b9ab9d8265b90b03b): perf score=0.033370
I20240501 19:58:50.874184 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(0fb481d394034a8f85af613beb5c9ec8) complete. Timing: real 0.023s user 0.015s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":1420,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1198,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":109,"threads_started":1,"wal-append.queue_time_us":191}
I20240501 19:58:50.875078 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(22b272a8582d4060a6a527ac4fce9ae5): 4834 bytes on disk
I20240501 19:58:50.875641 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(22b272a8582d4060a6a527ac4fce9ae5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 19:58:50.876364 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(db240ec5f8b2444fb6114ad814f3cdb7): perf score=0.033386
I20240501 19:58:50.884289 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(04ccc8addc55441b9ab9d8265b90b03b) complete. Timing: real 0.017s user 0.015s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1180,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":555,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":182}
I20240501 19:58:50.885205 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(745e39ec23db4f90bb8b1e8b49a50244): 4921 bytes on disk
I20240501 19:58:50.885519 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(745e39ec23db4f90bb8b1e8b49a50244) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 19:58:50.886410 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(3e84a3865ce64eaebad4df46ec01842d): perf score=0.033371
I20240501 19:58:50.900828 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(db240ec5f8b2444fb6114ad814f3cdb7) complete. Timing: real 0.024s user 0.014s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1564,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":557,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":104,"threads_started":1,"wal-append.queue_time_us":181}
I20240501 19:58:50.901796 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(d940b16d07d54945abe4a10a11dd6a2a): 4582 bytes on disk
I20240501 19:58:50.902208 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(d940b16d07d54945abe4a10a11dd6a2a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 19:58:50.902153 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(3e84a3865ce64eaebad4df46ec01842d) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1102,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":505,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":114,"threads_started":1,"wal-append.queue_time_us":190}
I20240501 19:58:50.902864 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(3e84a3865ce64eaebad4df46ec01842d): 4921 bytes on disk
I20240501 19:58:50.903002 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(0fb481d394034a8f85af613beb5c9ec8): 4581 bytes on disk
I20240501 19:58:50.903236 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(3e84a3865ce64eaebad4df46ec01842d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 19:58:50.903589 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(0fb481d394034a8f85af613beb5c9ec8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4}
I20240501 19:58:50.904018 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(04ccc8addc55441b9ab9d8265b90b03b): 4579 bytes on disk
I20240501 19:58:50.904166 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(db240ec5f8b2444fb6114ad814f3cdb7): 4581 bytes on disk
I20240501 19:58:50.904351 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(04ccc8addc55441b9ab9d8265b90b03b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 19:58:50.904433 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(db240ec5f8b2444fb6114ad814f3cdb7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240501 19:58:50.909082 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(04a70a2ee2144f2287795ffe723d2716): perf score=0.033392
I20240501 19:58:50.924041 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(04a70a2ee2144f2287795ffe723d2716) complete. Timing: real 0.015s user 0.013s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1371,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":459,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":103,"threads_started":1,"wal-append.queue_time_us":178}
I20240501 19:58:50.924934 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(851eeb0ab3364529907d9d09f2a4022e): perf score=0.033376
I20240501 19:58:50.936959 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(851eeb0ab3364529907d9d09f2a4022e) complete. Timing: real 0.012s user 0.007s sys 0.003s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1222,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":421,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":99,"threads_started":1,"wal-append.queue_time_us":182}
I20240501 19:58:50.938014 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(04a70a2ee2144f2287795ffe723d2716): 4834 bytes on disk
I20240501 19:58:50.938410 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(04a70a2ee2144f2287795ffe723d2716) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 19:58:50.939214 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(851eeb0ab3364529907d9d09f2a4022e): 4834 bytes on disk
I20240501 19:58:50.939522 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(851eeb0ab3364529907d9d09f2a4022e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240501 19:58:51.155584 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(cf6948589837482a9f580731e3b861b8): perf score=0.033359
I20240501 19:58:51.170562 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(cf6948589837482a9f580731e3b861b8) complete. Timing: real 0.015s user 0.010s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":141,"dirs.run_wall_time_us":1499,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":494,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":131,"threads_started":1,"wal-append.queue_time_us":199}
I20240501 19:58:51.171378 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(cf6948589837482a9f580731e3b861b8): 4834 bytes on disk
I20240501 19:58:51.171744 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(cf6948589837482a9f580731e3b861b8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 19:58:51.190968 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(32a17c2e392a49b7b181fa7a25a778a1): perf score=0.033353
I20240501 19:58:51.207020 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(32a17c2e392a49b7b181fa7a25a778a1) complete. Timing: real 0.016s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1399,"drs_written":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4,"lbm_write_time_us":451,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":72,"threads_started":1,"wal-append.queue_time_us":164}
I20240501 19:58:51.207721 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(32a17c2e392a49b7b181fa7a25a778a1): 4921 bytes on disk
I20240501 19:58:51.208039 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(32a17c2e392a49b7b181fa7a25a778a1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 19:58:51.297479 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:58:51.297883 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:51.423075 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(f08535cb2f46464b9e510a45fee67b4f): perf score=0.033391
I20240501 19:58:51.434952 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(f08535cb2f46464b9e510a45fee67b4f) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":45,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1294,"drs_written":1,"lbm_read_time_us":34,"lbm_reads_lt_1ms":4,"lbm_write_time_us":368,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":89,"threads_started":1,"wal-append.queue_time_us":174}
I20240501 19:58:51.435684 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(f08535cb2f46464b9e510a45fee67b4f): 4922 bytes on disk
I20240501 19:58:51.436017 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(f08535cb2f46464b9e510a45fee67b4f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240501 19:58:51.657431 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:58:51.669636 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:58:51.709846 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(8bc9fe9ade8246779164d6db31b11e16): perf score=0.033349
I20240501 19:58:51.722103 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(8bc9fe9ade8246779164d6db31b11e16) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":165,"dirs.run_cpu_time_us":170,"dirs.run_wall_time_us":1389,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":380,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":150}
I20240501 19:58:51.722815 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(8bc9fe9ade8246779164d6db31b11e16): 4921 bytes on disk
I20240501 19:58:51.723132 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(8bc9fe9ade8246779164d6db31b11e16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
W20240501 19:58:51.747457 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:58:51.887526 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:52.069685 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 19:58:52.158377 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(544caa38514643d5b81cd6ed85e0bae7): perf score=0.033397
I20240501 19:58:52.172179 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(544caa38514643d5b81cd6ed85e0bae7) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":148,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1527,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":465,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":132,"threads_started":2,"wal-append.queue_time_us":122}
I20240501 19:58:52.172875 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(955edb7c12fa4b32ac58827f131931e3): perf score=0.033394
I20240501 19:58:52.183041 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(955edb7c12fa4b32ac58827f131931e3) complete. Timing: real 0.010s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":62,"dirs.run_cpu_time_us":153,"dirs.run_wall_time_us":1045,"drs_written":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4,"lbm_write_time_us":360,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":59,"threads_started":1,"wal-append.queue_time_us":121}
I20240501 19:58:52.183832 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(544caa38514643d5b81cd6ed85e0bae7): 4581 bytes on disk
I20240501 19:58:52.184151 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(544caa38514643d5b81cd6ed85e0bae7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4}
I20240501 19:58:52.184648 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(ce6add2f19a649b3a12497951047705c): perf score=0.033363
I20240501 19:58:52.194556 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(ce6add2f19a649b3a12497951047705c) complete. Timing: real 0.010s user 0.005s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":164,"dirs.run_wall_time_us":1190,"drs_written":1,"lbm_read_time_us":31,"lbm_reads_lt_1ms":4,"lbm_write_time_us":361,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":57,"threads_started":1,"wal-append.queue_time_us":121}
I20240501 19:58:52.195113 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(955edb7c12fa4b32ac58827f131931e3): 4581 bytes on disk
I20240501 19:58:52.195461 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(955edb7c12fa4b32ac58827f131931e3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 19:58:52.196018 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(ce6add2f19a649b3a12497951047705c): 4581 bytes on disk
I20240501 19:58:52.196384 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(ce6add2f19a649b3a12497951047705c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 19:58:52.224958 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(e4b15761177f4cffb7aa5e4d16e42ac7): perf score=0.033385
I20240501 19:58:52.240836 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(e4b15761177f4cffb7aa5e4d16e42ac7) complete. Timing: real 0.016s user 0.011s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":210,"dirs.run_cpu_time_us":163,"dirs.run_wall_time_us":3975,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":493,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":140,"threads_started":2,"wal-append.queue_time_us":111}
I20240501 19:58:52.241627 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(e4b15761177f4cffb7aa5e4d16e42ac7): 4581 bytes on disk
I20240501 19:58:52.241967 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(e4b15761177f4cffb7aa5e4d16e42ac7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
I20240501 19:58:52.242585 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(7e4d79f5e11b4923acb2ce543884b54a): perf score=0.033371
I20240501 19:58:52.253743 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(7e4d79f5e11b4923acb2ce543884b54a) complete. Timing: real 0.011s user 0.004s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1124,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":399,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":59,"threads_started":1,"wal-append.queue_time_us":137}
I20240501 19:58:52.254446 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(7e4d79f5e11b4923acb2ce543884b54a): 4834 bytes on disk
I20240501 19:58:52.254822 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(7e4d79f5e11b4923acb2ce543884b54a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 19:58:52.301955 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:52.337297 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:58:52.382591 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:58:52.447458 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(0bf64d9f3ce94db28b032cdcd795a255): perf score=0.033366
I20240501 19:58:52.458657 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(0bf64d9f3ce94db28b032cdcd795a255) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1352,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":382,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":79,"threads_started":1,"wal-append.queue_time_us":141}
I20240501 19:58:52.459488 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(0bf64d9f3ce94db28b032cdcd795a255): 4921 bytes on disk
I20240501 19:58:52.459800 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(0bf64d9f3ce94db28b032cdcd795a255) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 19:58:52.505977 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(0bf64d9f3ce94db28b032cdcd795a255): perf score=0.033383
I20240501 19:58:52.517199 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(0bf64d9f3ce94db28b032cdcd795a255) complete. Timing: real 0.011s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":74,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1416,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":398,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":66,"threads_started":1,"wal-append.queue_time_us":146}
I20240501 19:58:52.517942 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(0bf64d9f3ce94db28b032cdcd795a255): 4921 bytes on disk
I20240501 19:58:52.518299 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(0bf64d9f3ce94db28b032cdcd795a255) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4}
W20240501 19:58:52.798164 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:58:52.885489 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:52.906972 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:58:52.961601 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(745e39ec23db4f90bb8b1e8b49a50244): perf score=0.033376
I20240501 19:58:52.972708 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(745e39ec23db4f90bb8b1e8b49a50244) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1419,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":393,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":123}
I20240501 19:58:52.973471 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(04ccc8addc55441b9ab9d8265b90b03b): perf score=0.033342
I20240501 19:58:52.984316 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(04ccc8addc55441b9ab9d8265b90b03b) complete. Timing: real 0.011s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":88,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":1242,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":367,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":116,"threads_started":1,"wal-append.queue_time_us":176}
I20240501 19:58:52.985016 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(745e39ec23db4f90bb8b1e8b49a50244): 4921 bytes on disk
I20240501 19:58:52.985388 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(745e39ec23db4f90bb8b1e8b49a50244) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 19:58:52.985994 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(04ccc8addc55441b9ab9d8265b90b03b): 4579 bytes on disk
I20240501 19:58:52.986300 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(04ccc8addc55441b9ab9d8265b90b03b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240501 19:58:53.190954 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(0fb481d394034a8f85af613beb5c9ec8): perf score=0.033366
I20240501 19:58:53.202322 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(0fb481d394034a8f85af613beb5c9ec8) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":139,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1459,"drs_written":1,"lbm_read_time_us":36,"lbm_reads_lt_1ms":4,"lbm_write_time_us":442,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":129,"threads_started":2,"wal-append.queue_time_us":123}
I20240501 19:58:53.203044 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(0fb481d394034a8f85af613beb5c9ec8): 4581 bytes on disk
I20240501 19:58:53.203379 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(0fb481d394034a8f85af613beb5c9ec8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:58:53.204006 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(7d7ea564ea504163a1e17561be9a5130): perf score=0.033357
I20240501 19:58:53.215257 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(7d7ea564ea504163a1e17561be9a5130) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1185,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":345,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":141}
I20240501 19:58:53.215929 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(7d7ea564ea504163a1e17561be9a5130): 4834 bytes on disk
I20240501 19:58:53.216220 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(7d7ea564ea504163a1e17561be9a5130) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240501 19:58:53.216768 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(61bc22cdc6ab45529066fe6f4d14d4c7): perf score=0.033351
I20240501 19:58:53.228194 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(61bc22cdc6ab45529066fe6f4d14d4c7) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":66,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1166,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":359,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":69,"threads_started":1,"wal-append.queue_time_us":138}
I20240501 19:58:53.228838 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(61bc22cdc6ab45529066fe6f4d14d4c7): 4830 bytes on disk
I20240501 19:58:53.229208 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(61bc22cdc6ab45529066fe6f4d14d4c7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4}
I20240501 19:58:53.237653 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(d75bd8f836b947248ee124f456c7d721): perf score=0.033370
I20240501 19:58:53.248898 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(d75bd8f836b947248ee124f456c7d721) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1281,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":372,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":63,"threads_started":1,"wal-append.queue_time_us":147}
I20240501 19:58:53.249637 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(d75bd8f836b947248ee124f456c7d721): 4834 bytes on disk
I20240501 19:58:53.249974 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(d75bd8f836b947248ee124f456c7d721) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 19:58:53.270925 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(12a50be843fb49ee8a66c9a17d271d61): perf score=0.033380
I20240501 19:58:53.285611 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(12a50be843fb49ee8a66c9a17d271d61) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":142,"dirs.run_cpu_time_us":217,"dirs.run_wall_time_us":1432,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":413,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":111,"threads_started":2,"wal-append.queue_time_us":109}
I20240501 19:58:53.286321 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(d940b16d07d54945abe4a10a11dd6a2a): perf score=0.033375
I20240501 19:58:53.296554 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(d940b16d07d54945abe4a10a11dd6a2a) complete. Timing: real 0.010s user 0.009s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":37,"dirs.run_cpu_time_us":140,"dirs.run_wall_time_us":1109,"drs_written":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4,"lbm_write_time_us":347,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":58,"threads_started":1,"wal-append.queue_time_us":156}
I20240501 19:58:53.297233 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(12a50be843fb49ee8a66c9a17d271d61): 4921 bytes on disk
I20240501 19:58:53.297534 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(12a50be843fb49ee8a66c9a17d271d61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:58:53.298130 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(d940b16d07d54945abe4a10a11dd6a2a): 4582 bytes on disk
I20240501 19:58:53.298408 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(d940b16d07d54945abe4a10a11dd6a2a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
I20240501 19:58:53.299021 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(800d5f43cd1b4fb6bd8bbcac262aeb80): perf score=0.033347
I20240501 19:58:53.310367 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(800d5f43cd1b4fb6bd8bbcac262aeb80) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1109,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":350,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":53,"threads_started":1,"wal-append.queue_time_us":173}
I20240501 19:58:53.310979 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(800d5f43cd1b4fb6bd8bbcac262aeb80): 4920 bytes on disk
I20240501 19:58:53.311295 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(800d5f43cd1b4fb6bd8bbcac262aeb80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 19:58:53.317502 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:53.381983 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 19:58:53.480432 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(557994658e2c4c8bbc1eeacdab958ddd): perf score=0.033377
I20240501 19:58:53.497965 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(557994658e2c4c8bbc1eeacdab958ddd) complete. Timing: real 0.017s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1355,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":943,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":74,"threads_started":1,"wal-append.queue_time_us":136}
I20240501 19:58:53.498715 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(557994658e2c4c8bbc1eeacdab958ddd): 4921 bytes on disk
I20240501 19:58:53.499022 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(557994658e2c4c8bbc1eeacdab958ddd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
I20240501 19:58:53.499622 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(c7ddb6d79d384d39a709b5824f936d53): perf score=0.033376
I20240501 19:58:53.510404 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(c7ddb6d79d384d39a709b5824f936d53) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":142,"dirs.run_wall_time_us":1151,"drs_written":1,"lbm_read_time_us":32,"lbm_reads_lt_1ms":4,"lbm_write_time_us":371,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":75,"threads_started":1,"wal-append.queue_time_us":166}
I20240501 19:58:53.511018 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(c7ddb6d79d384d39a709b5824f936d53): 4834 bytes on disk
I20240501 19:58:53.511307 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(c7ddb6d79d384d39a709b5824f936d53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240501 19:58:53.562347 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(ca3285abe3654723a6922672391e9423): perf score=0.033391
I20240501 19:58:53.573091 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(ca3285abe3654723a6922672391e9423) complete. Timing: real 0.011s user 0.006s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":35,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1406,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":431,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":58,"threads_started":1,"wal-append.queue_time_us":129}
I20240501 19:58:53.573769 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(ca3285abe3654723a6922672391e9423): 4581 bytes on disk
I20240501 19:58:53.574120 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(ca3285abe3654723a6922672391e9423) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4}
W20240501 19:58:53.764444 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:53.782582 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:58:53.956609 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:58:54.066669 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:58:54.120280 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:58:54.246845 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 19:58:54.252789 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(e4b15761177f4cffb7aa5e4d16e42ac7): perf score=0.033380
I20240501 19:58:54.265091 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(e4b15761177f4cffb7aa5e4d16e42ac7) complete. Timing: real 0.012s user 0.010s sys 0.002s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":179,"dirs.run_cpu_time_us":124,"dirs.run_wall_time_us":1331,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":406,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":130,"threads_started":2,"wal-append.queue_time_us":122}
I20240501 19:58:54.265924 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(e4b15761177f4cffb7aa5e4d16e42ac7): 4581 bytes on disk
I20240501 19:58:54.266247 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(e4b15761177f4cffb7aa5e4d16e42ac7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
W20240501 19:58:54.304694 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:54.336537 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 19:58:54.577225 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling FlushMRSOp(955edb7c12fa4b32ac58827f131931e3): perf score=0.033388
W20240501 19:58:54.585820 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:58:54.588377 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: FlushMRSOp(955edb7c12fa4b32ac58827f131931e3) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1356,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":389,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":143,"threads_started":2,"wal-append.queue_time_us":121}
I20240501 19:58:54.589010 14596 maintenance_manager.cc:419] P 1d49e80be3d04fb99b711c62d61bac09: Scheduling UndoDeltaBlockGCOp(955edb7c12fa4b32ac58827f131931e3): 4581 bytes on disk
I20240501 19:58:54.589365 14486 maintenance_manager.cc:643] P 1d49e80be3d04fb99b711c62d61bac09: UndoDeltaBlockGCOp(955edb7c12fa4b32ac58827f131931e3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":42,"lbm_reads_lt_1ms":4}
W20240501 19:58:54.614459 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 19:58:54.743291 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:54.913849 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 19:58:55.015250 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling FlushMRSOp(db240ec5f8b2444fb6114ad814f3cdb7): perf score=0.033379
I20240501 19:58:55.027223 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: FlushMRSOp(db240ec5f8b2444fb6114ad814f3cdb7) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":176,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1509,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":412,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"spinlock_wait_cycles":896,"thread_start_us":124,"threads_started":2,"wal-append.queue_time_us":124}
I20240501 19:58:55.027909 14333 maintenance_manager.cc:419] P 9069ec6fcdb248479037f2d37e0e851d: Scheduling UndoDeltaBlockGCOp(db240ec5f8b2444fb6114ad814f3cdb7): 4581 bytes on disk
I20240501 19:58:55.028278 14223 maintenance_manager.cc:643] P 9069ec6fcdb248479037f2d37e0e851d: UndoDeltaBlockGCOp(db240ec5f8b2444fb6114ad814f3cdb7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
W20240501 19:58:55.037546 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:55.140257 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:58:55.242463 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:58:55.243050 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:55.306365 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(544caa38514643d5b81cd6ed85e0bae7): perf score=0.033391
I20240501 19:58:55.318223 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(544caa38514643d5b81cd6ed85e0bae7) complete. Timing: real 0.012s user 0.010s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":166,"dirs.run_cpu_time_us":196,"dirs.run_wall_time_us":1440,"drs_written":1,"lbm_read_time_us":37,"lbm_reads_lt_1ms":4,"lbm_write_time_us":405,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":197,"threads_started":2,"wal-append.queue_time_us":189}
I20240501 19:58:55.319072 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(544caa38514643d5b81cd6ed85e0bae7): 4581 bytes on disk
I20240501 19:58:55.319500 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(544caa38514643d5b81cd6ed85e0bae7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 19:58:55.520339 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling FlushMRSOp(557994658e2c4c8bbc1eeacdab958ddd): perf score=0.033347
I20240501 19:58:55.535933 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: FlushMRSOp(557994658e2c4c8bbc1eeacdab958ddd) complete. Timing: real 0.015s user 0.014s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":149,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1479,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":647,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":126,"threads_started":2,"wal-append.queue_time_us":112}
I20240501 19:58:55.536720 14727 maintenance_manager.cc:419] P c6d2c7096e404ea9a6f73b123d74c622: Scheduling UndoDeltaBlockGCOp(557994658e2c4c8bbc1eeacdab958ddd): 4921 bytes on disk
I20240501 19:58:55.537026 14617 maintenance_manager.cc:643] P c6d2c7096e404ea9a6f73b123d74c622: UndoDeltaBlockGCOp(557994658e2c4c8bbc1eeacdab958ddd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":43,"lbm_reads_lt_1ms":4}
I20240501 19:58:55.622977 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(8b61e94fc7e541c8aa84957560508a31): perf score=0.033356
I20240501 19:58:55.635051 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(8b61e94fc7e541c8aa84957560508a31) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":164,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":1480,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":421,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":122,"threads_started":2,"wal-append.queue_time_us":124}
I20240501 19:58:55.635782 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(8b61e94fc7e541c8aa84957560508a31): 4921 bytes on disk
I20240501 19:58:55.636109 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(8b61e94fc7e541c8aa84957560508a31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":44,"lbm_reads_lt_1ms":4}
W20240501 19:58:55.739984 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:58:55.747646 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:58:55.778513 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 19:58:55.821424 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling FlushMRSOp(745e39ec23db4f90bb8b1e8b49a50244): perf score=0.033370
I20240501 19:58:55.834014 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: FlushMRSOp(745e39ec23db4f90bb8b1e8b49a50244) complete. Timing: real 0.012s user 0.011s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":154,"dirs.run_cpu_time_us":154,"dirs.run_wall_time_us":1564,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":412,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":127,"threads_started":2,"wal-append.queue_time_us":132}
I20240501 19:58:55.834683 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(745e39ec23db4f90bb8b1e8b49a50244): 4921 bytes on disk
I20240501 19:58:55.835005 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(745e39ec23db4f90bb8b1e8b49a50244) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4}
W20240501 19:58:55.843327 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:58:55.843952 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 19:58:55.887326 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(8bc9fe9ade8246779164d6db31b11e16): perf score=0.033365
I20240501 19:58:55.898959 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(8bc9fe9ade8246779164d6db31b11e16) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":302,"dirs.run_wall_time_us":1632,"drs_written":1,"lbm_read_time_us":33,"lbm_reads_lt_1ms":4,"lbm_write_time_us":402,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":64,"threads_started":1,"wal-append.queue_time_us":135}
I20240501 19:58:55.899677 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(8bc9fe9ade8246779164d6db31b11e16): 4921 bytes on disk
I20240501 19:58:55.900025 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(8bc9fe9ade8246779164d6db31b11e16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4}
I20240501 19:58:56.151340 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(ae2885f7933f462c96db9a99a4eb2090): perf score=0.033391
I20240501 19:58:56.164796 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(ae2885f7933f462c96db9a99a4eb2090) complete. Timing: real 0.013s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1530,"drs_written":1,"lbm_read_time_us":35,"lbm_reads_lt_1ms":4,"lbm_write_time_us":488,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":68,"threads_started":1,"wal-append.queue_time_us":150}
I20240501 19:58:56.165638 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling FlushMRSOp(7e4d79f5e11b4923acb2ce543884b54a): perf score=0.033341
I20240501 19:58:56.179924 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: FlushMRSOp(7e4d79f5e11b4923acb2ce543884b54a) complete. Timing: real 0.014s user 0.012s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1313,"drs_written":1,"lbm_read_time_us":38,"lbm_reads_lt_1ms":4,"lbm_write_time_us":564,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":77,"threads_started":1,"wal-append.queue_time_us":190}
I20240501 19:58:56.180686 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(7e4d79f5e11b4923acb2ce543884b54a): 4834 bytes on disk
I20240501 19:58:56.181027 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(7e4d79f5e11b4923acb2ce543884b54a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4}
I20240501 19:58:56.181622 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(ae2885f7933f462c96db9a99a4eb2090): 4834 bytes on disk
I20240501 19:58:56.181900 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(ae2885f7933f462c96db9a99a4eb2090) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":41,"lbm_reads_lt_1ms":4}
W20240501 19:58:56.204420 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:56.278153 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:58:56.533506 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:58:56.569558 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:58:56.826206 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:56.873519 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:58:56.922394 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:58:57.195672 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:58:57.283875 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:57.367910 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 19:58:57.662856 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:58:57.817373 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:58.241384 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:58.294433 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:58:58.401183 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:58:58.697372 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:58.737335 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:58:58.964553 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:58:59.258430 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:59.321844 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 19:58:59.322412 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:58:59.335649 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:58:59.343425 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 19:58:59.456168 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:58:59.651927 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:58:59.662457 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:58:59.811674 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 19:59:00.204037 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:00.206112 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:00.269896 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:00.305368 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:00.629587 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:59:00.795435 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:00.797242 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:59:00.807400 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:01.010495 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:59:01.344449 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:01.434700 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:59:01.506134 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:59:01.525537 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:01.847684 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:01.998466 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:59:02.206241 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:59:02.237859 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:02.257409 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:02.389585 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 19:59:02.699856 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:02.824684 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:02.930150 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:02.930874 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:02.968025 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:03.007745 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:59:03.296414 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:03.765656 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:03.771900 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:04.136461 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:04.194540 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:04.291489 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:04.479462 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:59:04.640405 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:04.687168 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:04.693555 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:04.793851 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:04.838478 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:04.976840 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 19:59:05.175917 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:05.266829 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:05.527768 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:05.528335 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:05.575397 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:05.581692 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:05.841975 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:05.983543 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:06.299644 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:59:06.387197 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:06.516137 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:59:06.548487 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:06.598237 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:06.819756 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:07.147464 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:07.200125 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:07.262089 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 19:59:07.418201 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:07.450134 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:07.466840 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:07.789397 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:08.057451 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:08.163158 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:08.233745 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:08.574638 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:08.787425 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:08.850323 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:09.213196 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:09.243422 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:09.399014 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:09.519716 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 19:59:09.520424 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 19:59:09.755753 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:09.795881 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:09.802580 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:09.864225 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:10.066730 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:10.307113 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:10.631064 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 19:59:10.631781 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:10.710865 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:10.711475 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:10.758637 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:10.951854 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:11.131669 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:11.387352 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:11.570890 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:11.643505 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:11.680869 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:11.783226 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:11.857506 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:12.043524 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:12.223639 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 19:59:12.250403 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:12.510255 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:12.584933 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:12.794173 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:13.120012 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:13.404975 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:13.445355 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:13.452373 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:13.957120 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:14.021378 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:14.176035 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:14.210543 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:14.459582 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:14.623134 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:59:14.623755 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:14.892000 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:14.914455 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 19:59:15.030853 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:15.124018 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:15.127383 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:15.324507 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:15.546082 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:15.706580 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:15.725296 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:15.726040 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:15.928030 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:59:16.162902 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:16.229748 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:16.660840 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:16.666079 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:16.748535 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:16.838994 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:16.915982 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:17.021749 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:59:17.153445 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:17.203449 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:17.585922 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:17.599668 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 19:59:17.607741 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:17.734283 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:17.912501 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:18.059007 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:18.286901 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:18.670280 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:18.771540 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:19.147457 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:19.302470 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:19.308246 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:19.325253 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:19.447736 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:19.797125 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:19.885630 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 19:59:19.895045 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 19:59:20.053282 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:20.179697 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:20.195704 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:20.256561 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:20.340190 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:20.731395 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 19:59:20.732106 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:20.733366 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:21.004272 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:21.119455 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:21.169411 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:21.496414 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:21.715132 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:21.753079 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:21.795028 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:21.952641 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:22.175902 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:22.297991 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:22.417209 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 19:59:22.560223 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:59:22.598881 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:22.735317 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:22.742209 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:22.789920 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:23.327926 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:23.329990 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:23.674386 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:23.734041 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:23.896852 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:24.001894 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:24.302996 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:24.324697 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:24.483443 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:24.717803 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 19:59:24.800288 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:24.910262 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:24.946348 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 19:59:25.217571 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:25.286020 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:25.335492 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:25.523809 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:25.745354 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:25.746430 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:25.778589 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:26.169693 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:26.188239 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:26.264941 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:26.301198 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:26.401996 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:26.632153 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:26.807266 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:27.051203 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:27.064568 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:27.284380 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:27.431456 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 19:59:27.489063 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:27.629151 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:27.663496 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:27.868496 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:27.890200 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:28.002810 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:28.068027 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:28.534574 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:28.675890 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:28.732427 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:28.798118 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:29.135757 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:29.298570 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:29.446273 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:29.609061 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:29.794106 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:29.895987 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 19:59:29.956674 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 19:59:30.088522 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:30.216190 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:30.454380 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:30.602324 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:30.630528 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:30.634955 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:31.123453 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:31.151427 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:59:31.152168 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:31.182847 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:31.420974 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:31.599400 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:31.667513 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:59:31.806293 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:32.028409 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:32.123919 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:32.318816 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:32.575155 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:32.592170 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:32.686481 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 19:59:32.752123 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:32.821729 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:33.110586 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:33.247615 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:33.318861 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:33.657186 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:33.681962 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:33.726692 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:33.936687 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:34.130249 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:34.417130 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:34.713728 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:34.799569 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:34.898007 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:35.074774 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:59:35.075567 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 19:59:35.223773 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:35.301088 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:35.597467 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:35.627157 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:35.675412 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:35.877478 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:35.984494 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:36.140295 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:36.407166 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:36.455595 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:36.458338 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:36.753783 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:36.868389 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:36.931865 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:37.227468 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:37.358597 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:37.370081 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:37.425758 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:37.486032 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 19:59:37.730566 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:38.050276 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:38.101761 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:59:38.102334 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:38.107297 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:38.499934 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:38.612299 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:38.821693 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:38.857859 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:38.878695 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:39.097275 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:39.382730 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:39.635931 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:39.647301 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:39.746600 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:39.936102 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:40.152968 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:59:40.153674 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 19:59:40.216462 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:40.644680 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:40.665745 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:40.753777 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:40.767987 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:41.174577 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:41.264494 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:41.510176 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:41.531534 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:41.726569 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:41.773875 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:41.795270 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:41.994556 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:42.245127 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:42.251304 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:42.447853 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:42.479267 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:42.603492 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 19:59:42.851917 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:42.958726 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:43.093209 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:43.241851 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:43.354815 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:59:43.364118 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:43.813324 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:43.906973 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:44.006016 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:44.037441 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:44.135320 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:44.436333 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:44.462314 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:44.900470 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:44.902791 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:45.138626 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:45.190788 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 19:59:45.191376 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 19:59:45.331609 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:45.462132 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:45.679913 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:45.880165 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:46.057931 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:46.281762 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:46.426358 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:46.490470 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:46.614425 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:46.667057 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:46.780848 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:46.835529 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:47.127468 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:47.165158 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 19:59:47.171176 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:47.391949 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:47.616466 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 19:59:47.663780 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:47.789705 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:48.180078 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:48.257772 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:48.554674 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:48.663496 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:48.698604 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:48.832372 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:48.847332 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:59:49.053112 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:49.065021 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:49.292894 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:49.505692 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:49.518666 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:49.582280 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:49.854135 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:50.019804 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:50.224809 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:59:50.467352 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 19:59:50.594987 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:50.638424 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:50.930873 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:51.180351 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:51.193624 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:51.204061 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:51.254966 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:51.639603 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:51.694880 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:51.794646 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:51.927561 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 19:59:52.032996 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:52.219754 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:52.306746 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:52.370239 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:52.653122 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:52.991101 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:53.079073 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 19:59:53.237138 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:53.420431 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:53.599787 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:53.602049 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:53.845774 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 19:59:53.879388 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 19:59:53.971779 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:54.114596 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:54.132028 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:54.307777 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 19:59:54.311019 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 19:59:54.482965 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:54.541222 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:55.161314 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:55.237231 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:55.291339 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 19:59:55.291886 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 19:59:55.783635 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:55.942600 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 19:59:55.979991 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 19:59:55.980675 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 19:59:56.098191 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 19:59:56.278545 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 19:59:56.294301 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:56.448261 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 19:59:56.698060 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:56.718163 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 19:59:56.873620 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:56.948091 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 19:59:57.343272 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:57.405476 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 19:59:57.406121 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:57.688175 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 19:59:57.739266 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 19:59:57.883172 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:58.293933 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:58.320103 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 19:59:58.364377 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 19:59:58.595458 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 19:59:58.645076 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 19:59:58.646389 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 19:59:58.694833 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 19:59:58.770187 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:58.921689 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 19:59:59.030354 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 19:59:59.184947 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:59.188844 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 19:59:59.557704 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 19:59:59.657698 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 19:59:59.851738 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:00:00.123412 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:00:00.152180 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:00.449553 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:00:00.615952 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:00.672135 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:00:00.762233 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:00:00.865176 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:00:01.046382 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:01.157282 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:01.183502 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:00:01.276314 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:00:01.394608 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:01.518517 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:01.833427 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:00:01.990758 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:00:02.076385 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:02.422901 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:02.552122 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:00:02.561898 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:02.599184 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:00:02.628036 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:02.944723 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:02.945456 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:02.977782 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:00:03.213147 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:03.426833 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:03.637375 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:00:03.638028 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:00:03.662379 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:00:03.670598 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:03.786826 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:00:03.874629 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:04.104655 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:04.432691 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:00:04.517473 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20240501 20:00:04.619693 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:04.715786 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:04.953750 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:00:05.193814 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:05.279639 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:00:05.478402 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:00:05.683173 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:05.800527 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:05.901996 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:05.964773 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:06.059581 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:06.158509 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:06.324826 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:00:06.327541 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:06.405854 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:06.720463 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:06.853192 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:07.149057 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:07.353497 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:07.588814 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:07.636328 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:00:07.827891 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:07.828572 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:07.855885 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:08.279196 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:08.362785 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:08.447531 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:08.518296 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:08.676926 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:08.688282 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:08.821475 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:08.822127 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:08.965713 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:09.338335 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:09.422866 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:09.643095 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:09.773382 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:09.835886 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:10.161211 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:00:10.366392 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:10.455952 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:10.580776 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:10.614647 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:00:10.828626 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:11.018271 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:11.056901 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:11.140318 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:11.178304 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:11.247388 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:11.371886 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:11.617025 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:11.741531 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:12.085666 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:12.493830 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:12.510066 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:12.582156 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:12.691030 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:00:12.910210 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:13.017441 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:13.018589 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:13.063889 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:13.227963 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:13.396406 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:13.689390 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:13.692781 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:13.698911 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:13.718853 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:13.878435 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:13.973778 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:14.188390 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:00:14.316201 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:14.668488 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:14.915303 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:14.945482 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:15.011566 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:00:15.365211 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:15.385838 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:15.578652 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:15.620715 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:15.759366 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:00:15.856177 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:16.060111 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:16.068742 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:16.088502 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:16.343654 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:16.388698 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:16.495102 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:16.495791 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:16.804690 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:17.284626 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:00:17.288262 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:17.412267 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:17.578485 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:17.723189 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:17.924377 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:18.183319 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:18.190655 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:18.286901 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:18.304735 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:18.417143 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:18.496325 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:18.698720 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:18.888854 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:18.927255 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:19.036718 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:19.331086 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:19.470249 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:00:19.713171 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:19.857461 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:19.941612 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:00:20.012869 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:20.168879 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:00:20.332309 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:20.657068 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:20.726214 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:20.797917 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:20.901729 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:21.000999 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:21.023620 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:00:21.107975 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:21.186317 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:21.235651 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:21.519109 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:21.661798 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:21.766278 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:22.222308 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:22.525848 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:22.615849 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:00:22.716146 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:22.820709 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:23.040375 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:23.097327 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:23.341645 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:23.399268 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:23.400027 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:23.420325 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:23.502768 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:23.525465 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:23.684545 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:23.907393 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:24.316262 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:24.393435 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:24.666895 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:00:24.904129 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:25.086477 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:25.158257 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:25.236129 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:00:25.451988 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:25.615398 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:00:25.615958 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:25.761195 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:25.763235 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:25.977877 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:26.063561 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:00:26.127451 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:26.236742 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:26.390945 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:26.423807 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:26.655269 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:26.767884 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:26.786899 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:27.294845 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:27.356357 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:27.404995 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:27.698084 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:00:27.774761 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:28.299808 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:28.320463 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:28.346680 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:28.379529 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:00:28.417892 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:28.815388 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:28.816042 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:28.871827 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:29.018679 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:29.109575 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:29.200543 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:29.226454 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:29.624301 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:29.815311 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:29.892505 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:30.007258 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:30.097687 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:30.262391 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:00:30.598594 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:30.664984 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:30.762156 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:30.762722 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:31.059381 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:31.070614 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:31.165225 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:00:31.460961 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:31.590945 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:31.601172 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:31.768950 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:31.808831 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:31.859911 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:31.998258 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:32.316944 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:32.425663 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:32.499439 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:32.840320 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:32.944640 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:00:33.096145 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:33.183068 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:33.319589 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:33.427676 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:33.773687 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:33.830475 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:33.831086 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:33.843449 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:34.329401 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:34.344533 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:34.528990 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:34.547322 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:34.603032 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:34.863467 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:34.901134 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:35.210950 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:35.469342 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:35.508447 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:00:35.697726 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:35.880935 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:35.881757 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:36.063076 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:36.231129 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:36.315901 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:36.318080 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:00:36.532255 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:36.755125 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:36.894022 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:37.038283 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:37.105437 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:37.160001 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:37.160759 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:37.514432 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:37.617334 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:37.992815 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:38.190446 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:00:38.349092 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:38.444917 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:38.471275 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:38.501227 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:38.603329 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:38.729434 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:38.985733 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:38.986418 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:39.361513 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:39.398459 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:39.557271 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:39.679246 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:39.752552 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:39.858237 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:39.938103 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:40.276862 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:40.509745 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:40.653139 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:00:40.891819 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:40.939342 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:40.940057 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:41.005059 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:41.040244 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:41.396056 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:41.422865 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:41.602567 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:00:41.698011 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:41.846302 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:42.072178 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:42.112629 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:42.231050 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:42.260291 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:42.271848 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:42.343151 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:42.650065 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:43.132787 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:00:43.153144 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:43.240000 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:43.429555 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:43.632367 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:43.648780 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:00:43.690425 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:43.696713 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:44.018114 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:44.214203 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:44.267589 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:44.628593 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:44.754426 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:44.770872 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:44.850119 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:45.014981 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:45.351234 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:45.537112 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:45.623982 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:00:45.801349 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:45.802631 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:00:45.972798 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:46.040223 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:46.040961 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:46.041450 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:46.395529 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:46.655148 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:00:46.782328 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:46.944509 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:47.056414 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:47.167565 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:47.261858 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:47.326987 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:47.343017 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:47.542901 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:00:47.863754 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:48.008812 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:48.418344 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:48.518754 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:00:48.551780 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:48.563565 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:48.654325 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:48.965616 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:49.010021 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:00:49.010639 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:49.087795 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:49.417064 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:49.497805 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:49.649549 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:49.874238 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:49.928478 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:49.991745 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:50.318713 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:50.469715 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:50.785259 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:50.795083 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:50.944509 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:51.055051 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:51.078397 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:00:51.250603 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:51.282968 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:51.360653 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:00:51.700569 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:51.816427 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:51.855535 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:00:51.856225 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:52.049160 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:52.258512 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:52.601049 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:52.601755 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:52.681913 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:00:52.837572 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:52.998533 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:00:53.259980 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:53.423059 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:00:53.440570 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:53.793684 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:53.834897 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:53.927484 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:54.097764 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:54.103503 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:00:54.248344 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:00:54.308182 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:54.408478 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:54.531323 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:55.015028 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:55.067013 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:55.081538 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:00:55.486294 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:55.636835 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:00:55.658159 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:00:55.976051 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:00:55.982887 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:00:56.005209 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:56.172324 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:00:56.524343 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:56.557740 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:56.765390 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:00:56.765925 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:00:56.831454 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:00:56.983017 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:00:57.021636 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:00:57.100124 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:57.575596 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:57.594326 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:00:57.729552 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:00:57.974938 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:58.111740 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:58.112442 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:00:58.268060 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:00:58.387775 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:00:58.449649 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:00:58.561450 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:59.084908 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:00:59.122589 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:59.217988 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:59.218612 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:00:59.269414 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:00:59.436729 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:00:59.508543 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:00:59.555804 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:00:59.927337 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:00.298738 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:00.305971 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:01:00.460122 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:00.619315 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:00.765347 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:01:00.841864 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:01:00.869464 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:01.077003 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:01.106982 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:01:01.284044 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:01.475338 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:01:01.604300 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:01.676666 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:01:01.695865 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:02.024478 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:02.031844 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:01:02.090356 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:01:02.175315 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:02.568172 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:02.578781 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:02.840310 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:01:02.972795 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:02.973513 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:02.996656 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:03.171686 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:03.217751 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:03.218427 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:03.368381 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:01:03.425850 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:03.465101 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:03.962919 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:03.990252 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:04.090509 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:04.399361 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:01:04.415362 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:04.448036 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:04.448637 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:04.751137 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:04.893784 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:04.971912 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:05.077802 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:05.332342 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:05.713532 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:05.733355 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:05.841500 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:01:05.864800 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:05.909276 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:06.150936 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:06.286201 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:06.340344 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:06.773089 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:06.802831 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:06.907414 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:07.015269 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:07.245397 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:07.273641 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:07.355505 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:07.528923 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:01:07.529639 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:07.868852 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:08.140663 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:08.146781 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:08.346451 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:08.380671 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:08.398844 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:01:08.484213 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:08.729205 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:08.967149 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:08.977406 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:09.574381 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:09.574965 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:09.588881 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:09.589534 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:09.637184 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:09.676096 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:10.064921 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:10.100589 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:10.573413 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:10.660687 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:10.682133 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:10.737982 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:10.858915 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:01:10.975122 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:11.294864 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:11.525990 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:11.526701 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:11.739326 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:11.997870 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:12.184018 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:12.325412 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:12.386291 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:12.585266 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:01:12.585915 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:12.615839 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:12.695700 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:13.033524 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:13.059089 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:13.100863 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:13.247915 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:13.399665 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:13.526280 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:01:13.531221 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:13.638046 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:13.862080 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:14.150041 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:14.226191 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:14.510238 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:14.700171 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:14.758775 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:14.759438 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:14.881381 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:15.229985 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:15.309202 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:15.585788 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:15.633455 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:15.667951 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:15.743841 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:15.748348 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:16.081611 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:01:16.090036 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:16.242010 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:16.611527 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:16.626672 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:16.781944 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:17.042874 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:17.154525 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:17.175482 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:17.589818 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:01:17.753144 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:17.864670 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:17.988130 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:18.061224 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:18.257220 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:18.312882 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:18.328441 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:18.456095 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:18.593726 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:01:18.701519 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:18.715512 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:18.784103 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:19.265980 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:19.316429 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:19.330816 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:19.453907 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:19.824548 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:01:19.850637 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:19.932953 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:20.165511 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:20.556694 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:20.588513 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:20.628868 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:20.860396 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:21.107818 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:21.210834 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:21.262842 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:01:21.415105 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:21.540517 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:21.627874 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:21.800084 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:21.846683 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:21.904218 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:22.167025 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:22.590831 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:01:22.764148 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:22.865204 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:22.875200 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 20:01:22.956893 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:23.090099 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:23.185513 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:23.369829 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:23.537583 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:01:23.670076 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:23.831250 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:23.833748 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:23.834287 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:23.889611 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:24.065557 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:24.381448 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:24.458549 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:24.585577 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:24.901762 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:25.038708 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:25.212524 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:25.246578 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:25.418093 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:25.602993 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:25.635887 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:25.760489 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:26.151857 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:26.181409 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:26.270924 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:01:26.287369 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:26.493912 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:26.560683 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:26.658802 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:27.086747 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:27.097883 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:27.247686 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:27.556284 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:27.659399 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:27.930591 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:27.990145 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:27.999601 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:28.092521 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:01:28.236582 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:28.442909 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:28.554994 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:01:28.678511 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:28.817028 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:28.912890 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:28.973672 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:28.987113 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:29.290592 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:29.315281 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:29.613134 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:29.739941 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:30.088060 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:30.155449 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:30.167836 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:30.262797 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:30.604676 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:30.643555 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:30.807796 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:31.035624 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:01:31.184928 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:31.225023 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:31.358932 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:31.392134 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:31.561749 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:31.709163 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:31.829490 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:32.245734 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:32.306794 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:32.322126 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:32.505874 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:32.711810 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:32.731676 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:33.153842 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:33.179908 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:01:33.254845 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:33.434424 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:33.593940 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:01:33.666370 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:33.740322 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:33.854167 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:33.961937 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:33.996158 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:34.108745 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:34.220777 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:34.301013 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:34.588752 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:34.637540 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:34.871213 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:35.151263 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:35.151921 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:35.378840 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:35.607857 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:35.885481 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:35.956600 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:36.051277 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:36.089792 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:01:36.186561 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:36.315380 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:36.473583 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:36.493397 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:36.504328 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:36.963390 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:36.992208 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:37.074433 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:37.383749 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:37.414911 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:37.534421 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:38.020017 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:38.055766 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:38.240979 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:01:38.405763 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:38.519430 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:38.608909 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:38.654633 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:01:38.674371 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:38.714735 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:38.715432 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:39.011512 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:39.013490 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:39.259799 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:39.595917 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:39.651597 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:39.710991 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:40.027977 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:40.032052 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:40.488782 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:40.508445 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:40.516139 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:40.894407 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:40.902530 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:41.130017 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:41.274909 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:41.278936 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:01:41.385255 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:41.408360 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:41.460669 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:41.510497 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:41.666853 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:42.062177 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:42.280412 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:42.620821 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:42.774538 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:42.922021 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:42.997058 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:43.034158 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:43.070063 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:43.331460 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:01:43.485270 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:43.675114 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:01:43.784704 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:43.889025 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:43.892221 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:43.908874 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:44.127110 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:44.127789 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:44.249259 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:44.425734 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:44.714653 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:44.843719 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:44.970875 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:45.476688 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:45.494177 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:45.501446 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:45.554064 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:01:45.554684 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:46.155437 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:46.196364 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:46.224035 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:46.226441 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:46.419116 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:46.480094 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:46.519292 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:01:46.775282 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:46.849931 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:47.264533 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:47.317852 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:47.729135 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:47.773747 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:01:47.937708 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:48.094794 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:48.165798 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:48.183306 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:48.475792 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:01:48.636917 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:01:48.722862 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:48.745668 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:49.030290 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:01:49.179421 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:49.195859 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240501 20:01:49.196602 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:49.295559 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:49.597350 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:49.638392 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:01:49.724129 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:49.750713 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:50.222654 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:50.372172 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:01:50.476845 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:01:50.623601 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:01:50.831291 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:50.831995 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:50.986954 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:01:51.261598 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:51.378542 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:01:51.391085 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:01:51.552467 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:01:51.598533 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:01:51.756446 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:51.855908 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:52.219905 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:01:52.353741 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:01:52.463845 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:52.957605 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:01:53.086267 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:53.121870 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:01:53.132354 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:01:53.410993 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:01:53.504921 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:53.509145 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:01:53.653481 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:53.866050 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:01:53.902899 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:53.920951 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:01:54.178704 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:01:54.231060 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:01:54.294816 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:01:54.507759 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:54.637112 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:01:54.896628 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:01:54.922343 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:01:55.109892 17833 consensus_queue.cc:579] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.373s)
W20240501 20:01:55.111287 12530 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 20:01:55.115945 12595 consensus_queue.cc:237] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 20:01:55.117359 14679 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:01:55.117373 12988 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 20:01:55.117802 17833 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:01:55.117985 18531 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 20:01:55.119793 18531 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:55.120216 12988 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:55.121490 14679 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:55.121876 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 22b272a8582d4060a6a527ac4fce9ae5 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:55.123234 12445 catalog_manager.cc:5484] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 7 to 8, NON_VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
W20240501 20:01:55.128113 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 20:01:55.128892 12595 consensus_queue.cc:237] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } }
W20240501 20:01:55.130019 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 20:01:55.130282 14679 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:01:55.130481 12988 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 20:01:55.130697 18531 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:01:55.130929 18530 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 20:01:55.131193 12531 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22b272a8582d4060a6a527ac4fce9ae5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.132519 17833 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 20:01:55.133190 12988 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 20:01:55.134171 12439 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 22b272a8582d4060a6a527ac4fce9ae5 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:55.134923 14679 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 20:01:55.135586 12447 catalog_manager.cc:5484] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 8 to 9, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: NON_VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: true } } }
I20240501 20:01:55.261698 12704 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:01:55.264209 12967 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.264245 14529 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 20:01:55.267099 14660 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.272001 12836 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.281316 14266 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.290555 12574 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:55.332937 14733 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskuLQw07/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 19:56:48.163777 14738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:48.164017 14738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:48.174997 14738 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:49.236428 14738 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 19:56:49.796629 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.796752 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.796829 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.796898 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.796969 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.797025 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.797080 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.797112 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.797145 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.797191 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.797236 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.797286 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.797333 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.797377 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.797441 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.797502 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.797549 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:49.851073 14738 rebalancer_tool.cc:1084] tablet e4b15761177f4cffb7aa5e4d16e42ac7: '58f750e4d9574c9ebfd3d9e57eefd411' -> '2ca06dad25a645bca71b09741798194a' move ignored: Illegal state: tablet e4b15761177f4cffb7aa5e4d16e42ac7: replica 2ca06dad25a645bca71b09741798194a is already present
I20240501 19:56:49.851189 14738 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:50.389624 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.389787 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.389952 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.390033 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.390076 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.390121 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.390159 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.390210 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.390254 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.390295 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.390337 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.390379 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:50.482724 14738 rebalancer_tool.cc:1074] tablet 8b61e94fc7e541c8aa84957560508a31: '2d52575d9e1b40dca0fc553d7295eb74' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:50.603010 14738 rebalancer_tool.cc:1074] tablet 7e4d79f5e11b4923acb2ce543884b54a: '58f750e4d9574c9ebfd3d9e57eefd411' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:50.706941 14738 rebalancer_tool.cc:1074] tablet 12a50be843fb49ee8a66c9a17d271d61: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:50.953382 14738 rebalancer_tool.cc:1074] tablet 955edb7c12fa4b32ac58827f131931e3: 'a238fc0c6d844cedb8adccb97d01ae5e' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:51.024116 14738 rebalancer_tool.cc:1074] tablet 8bc9fe9ade8246779164d6db31b11e16: '2d52575d9e1b40dca0fc553d7295eb74' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:51.266331 14738 rebalancer_tool.cc:1084] tablet 0bf64d9f3ce94db28b032cdcd795a255: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move ignored: Illegal state: tablet 0bf64d9f3ce94db28b032cdcd795a255: replica c6d2c7096e404ea9a6f73b123d74c622 is already present
I20240501 19:56:51.266435 14738 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:52.166911 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:52.167094 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:52.167171 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:52.167234 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:52.167295 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
I20240501 19:56:52.225453 14738 rebalancer_tool.cc:1074] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move scheduled
I20240501 19:56:52.323392 14738 rebalancer_tool.cc:1074] tablet 0fb481d394034a8f85af613beb5c9ec8: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:52.357829 14738 rebalancer_tool.cc:1074] tablet d940b16d07d54945abe4a10a11dd6a2a: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:52.442672 14738 rebalancer_tool.cc:1084] tablet 04ccc8addc55441b9ab9d8265b90b03b: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 19:56:52.442759 14738 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:53.142650 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:53.142843 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:53.356665 14738 rebalancer_tool.cc:1074] tablet e4b15761177f4cffb7aa5e4d16e42ac7: '2d52575d9e1b40dca0fc553d7295eb74' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:53.397168 14738 rebalancer_tool.cc:1084] tablet 0bf64d9f3ce94db28b032cdcd795a255: '58f750e4d9574c9ebfd3d9e57eefd411' -> '2ca06dad25a645bca71b09741798194a' move ignored: Not found: Peer with uuid 58f750e4d9574c9ebfd3d9e57eefd411 not found in consensus config
I20240501 19:56:53.397245 14738 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:54.314035 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2ca06dad25a645bca71b09741798194a to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:54.314548 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:54.314705 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server ebbbd9ac07a94e8dbbae6def0a2df415
I20240501 19:56:54.346002 14738 rebalancer_tool.cc:1074] tablet c7ddb6d79d384d39a709b5824f936d53: '2d52575d9e1b40dca0fc553d7295eb74' -> '' move scheduled
I20240501 19:56:54.370471 14738 rebalancer_tool.cc:1084] tablet 544caa38514643d5b81cd6ed85e0bae7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '58f750e4d9574c9ebfd3d9e57eefd411' move ignored: Illegal state: tablet 544caa38514643d5b81cd6ed85e0bae7: replica 58f750e4d9574c9ebfd3d9e57eefd411 is already present
I20240501 19:56:54.370548 14738 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:54.707535 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 9069ec6fcdb248479037f2d37e0e851d to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:54.708196 14738 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2d52575d9e1b40dca0fc553d7295eb74
I20240501 19:56:54.740770 14738 rebalancer_tool.cc:1074] tablet 22b272a8582d4060a6a527ac4fce9ae5: '2d52575d9e1b40dca0fc553d7295eb74' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:54.901293 14738 rebalancer_tool.cc:1136] tablet e4b15761177f4cffb7aa5e4d16e42ac7: '2d52575d9e1b40dca0fc553d7295eb74' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
I20240501 19:56:54.946544 14738 rebalancer_tool.cc:1136] tablet 0fb481d394034a8f85af613beb5c9ec8: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:55.121318 14738 rebalancer_tool.cc:1136] tablet 8b61e94fc7e541c8aa84957560508a31: '2d52575d9e1b40dca0fc553d7295eb74' -> '2ca06dad25a645bca71b09741798194a' move completed: Incomplete: tablet 8b61e94fc7e541c8aa84957560508a31, TS 2d52575d9e1b40dca0fc553d7295eb74 -> TS 2ca06dad25a645bca71b09741798194a move failed, target replica disappeared
I20240501 19:56:55.143846 14738 rebalancer_tool.cc:1136] tablet 12a50be843fb49ee8a66c9a17d271d61: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move completed: OK
I20240501 19:56:55.238978 14738 rebalancer_tool.cc:1136] tablet 955edb7c12fa4b32ac58827f131931e3: 'a238fc0c6d844cedb8adccb97d01ae5e' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
I20240501 19:56:55.383505 14738 rebalancer_tool.cc:1136] tablet d940b16d07d54945abe4a10a11dd6a2a: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 19:56:55.408345 14738 rebalancer_tool.cc:1123] tablet 8bc9fe9ade8246779164d6db31b11e16: '2d52575d9e1b40dca0fc553d7295eb74' -> '1d49e80be3d04fb99b711c62d61bac09' move is abandoned: Illegal state: 8bc9fe9ade8246779164d6db31b11e16: source replica 2d52575d9e1b40dca0fc553d7295eb74 does not have REPLACE attribute set
I20240501 19:56:55.609416 14738 rebalancer_tool.cc:1136] tablet 7e4d79f5e11b4923acb2ce543884b54a: '58f750e4d9574c9ebfd3d9e57eefd411' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:55.651705 14738 rebalancer_tool.cc:1136] tablet c7ddb6d79d384d39a709b5824f936d53: '2d52575d9e1b40dca0fc553d7295eb74' -> '' move completed: OK
I20240501 19:56:56.153357 14738 rebalancer_tool.cc:1136] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move completed: OK
I20240501 20:01:55.186877 14738 rebalancer_tool.cc:1136] tablet 22b272a8582d4060a6a527ac4fce9ae5: '2d52575d9e1b40dca0fc553d7295eb74' -> '2ca06dad25a645bca71b09741798194a' move completed: Incomplete: tablet 22b272a8582d4060a6a527ac4fce9ae5, TS 2d52575d9e1b40dca0fc553d7295eb74 -> TS 2ca06dad25a645bca71b09741798194a move failed, target replica disappeared
Illegal state: tablet server 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): unacceptable health status UNAVAILABLE
W20240501 20:01:55.367655 14615 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 20:01:55.418324 18455 consensus_queue.cc:579] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.367s)
W20240501 20:01:55.419642 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
I20240501 20:01:55.424257 12988 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 20:01:55.425827 14547 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
W20240501 20:01:55.426002 12921 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): Couldn't send request to peer a238fc0c6d844cedb8adccb97d01ae5e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.426231 18455 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 20:01:55.426832 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.428000 18611 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
I20240501 20:01:55.428416 14547 raft_consensus.cc:2932] T 851eeb0ab3364529907d9d09f2a4022e P 1d49e80be3d04fb99b711c62d61bac09 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
I20240501 20:01:55.429654 12440 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 851eeb0ab3364529907d9d09f2a4022e with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:55.430656 12448 catalog_manager.cc:5484] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 12 to 13, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:55.537979 18455 consensus_queue.cc:579] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.511s)
I20240501 20:01:55.540621 12988 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:01:55.542143 14286 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:01:55.542521 18612 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 20:01:55.542850 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.544348 18455 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } } }
W20240501 20:01:55.545329 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): Couldn't send request to peer ebbbd9ac07a94e8dbbae6def0a2df415. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.545786 12440 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 706878734216445f8422e422228d4230 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:55.546197 14286 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } } }
I20240501 20:01:55.546953 12452 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 12 to 13, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:55.622867 18634 ts_tablet_manager.cc:902] T 22b272a8582d4060a6a527ac4fce9ae5 P 1d49e80be3d04fb99b711c62d61bac09: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
F20240501 20:01:55.623064 18634 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-04a70a2ee2144f2287795ffe723d2716.queue_time_us, tablet-download-pool-04a70a2ee2144f2287795ffe723d2716.queue_time_us) (tablet-download-pool-04a70a2ee2144f2287795ffe723d2716.run_cpu_time_us, tablet-download-pool-04a70a2ee2144f2287795ffe723d2716.run_cpu_time_us) (tablet-download-pool-04a70a2ee2144f2287795ffe723d2716.run_wall_time_us, tablet-download-pool-04a70a2ee2144f2287795ffe723d2716.run_wall_time_us) (tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.queue_time_us, tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.queue_time_us) (tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.run_cpu_time_us, tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.run_cpu_time_us) (tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.run_wall_time_us, tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.run_wall_time_us) (tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.queue_time_us, tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.queue_time_us) (tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.run_cpu_time_us, tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.run_cpu_time_us) (tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.run_wall_time_us, tablet-download-pool-12a50be843fb49ee8a66c9a17d271d61.run_wall_time_us) (tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.queue_time_us, tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.queue_time_us) (tablet-download-pool-32a17c2e392a49b7b181fa7a25a778a1.queue_time_us, tablet-download-pool-32a17c2e392a49b7b181fa7a25a778a1.queue_time_us) (tablet-download-pool-32a17c2e392a49b7b181fa7a25a778a1.run_cpu_time_us, tablet-download-pool-32a17c2e392a49b7b181fa7a25a778a1.run_cpu_time_us) (tablet-download-pool-32a17c2e392a49b7b181fa7a25a778a1.run_wall_time_us, tablet-download-pool-32a17c2e392a49b7b181fa7a25a778a1.run_wall_time_us) (tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.queue_time_us, tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.queue_time_us) (tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.run_cpu_time_us, tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.run_cpu_time_us) (tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.run_wall_time_us, tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.run_wall_time_us) (tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.queue_time_us, tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.queue_time_us) (tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.run_cpu_time_us, tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.run_cpu_time_us) (tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.run_wall_time_us, tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.run_wall_time_us) (tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.queue_time_us, tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.queue_time_us) (tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.run_cpu_time_us, tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.run_cpu_time_us) (tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.run_wall_time_us, tablet-download-pool-851eeb0ab3364529907d9d09f2a4022e.run_wall_time_us) (tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.queue_time_us, tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.queue_time_us) (tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.run_cpu_time_us, tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.run_cpu_time_us) (tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.run_wall_time_us, tablet-download-pool-8b61e94fc7e541c8aa84957560508a31.run_wall_time_us) (tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.queue_time_us, tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.queue_time_us) (tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.run_cpu_time_us, tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.run_cpu_time_us) (tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.run_wall_time_us, tablet-download-pool-8bc9fe9ade8246779164d6db31b11e16.run_wall_time_us) (tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.queue_time_us, tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.queue_time_us) (tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.run_cpu_time_us, tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.run_cpu_time_us) (tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.run_wall_time_us, tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.run_wall_time_us) (tablet-download-pool-ca3285abe3654723a6922672391e9423.queue_time_us, tablet-download-pool-ca3285abe3654723a6922672391e9423.queue_time_us) (tablet-download-pool-ca3285abe3654723a6922672391e9423.run_cpu_time_us, tablet-download-pool-ca3285abe3654723a6922672391e9423.run_cpu_time_us) (tablet-download-pool-ca3285abe3654723a6922672391e9423.run_wall_time_us, tablet-download-pool-ca3285abe3654723a6922672391e9423.run_wall_time_us) (tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.queue_time_us, tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.queue_time_us) (tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.run_cpu_time_us, tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.run_cpu_time_us) (tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.run_wall_time_us, tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.run_wall_time_us) (tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.queue_time_us, tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.queue_time_us) (tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_cpu_time_us, tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_cpu_time_us) (tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_wall_time_us, tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_wall_time_us) (tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.queue_time_us, tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.queue_time_us) (tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.run_cpu_time_us, tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.run_cpu_time_us) (tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.run_wall_time_us, tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7fd5cb6d7dcd google::LogMessage::Fail() at ??:0
@ 0x7fd5cb6dbb93 google::LogMessage::SendToLog() at ??:0
@ 0x7fd5cb6d77cc google::LogMessage::Flush() at ??:0
@ 0x7fd5cb6d8f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fd5ccb41044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7fd5ccb25f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fd5ccb23efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7fd5cf6d677a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7fd5cf6dfa45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7fd5cf76076f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7fd5cf7744f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fd5cf75f226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7fd5cf76ec0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c5903fc75c std::function<>::operator()() at ??:0
@ 0x7fd5ccb28e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fd5ccb296ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fd5ccb2b03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55c5903fc75c std::function<>::operator()() at ??:0
@ 0x7fd5ccb18807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fd5cd3566db start_thread at ??:0
@ 0x7fd5ca88e71f clone at ??:0
I20240501 20:01:55.709653 17833 consensus_queue.cc:579] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.306s)
W20240501 20:01:55.712973 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:01:55.713335 12595 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 20:01:55.715174 14286 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 20:01:55.715750 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): Couldn't send request to peer a238fc0c6d844cedb8adccb97d01ae5e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7d7ea564ea504163a1e17561be9a5130. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.715925 18531 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:01:55.716802 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.717779 18531 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
I20240501 20:01:55.719661 12439 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7d7ea564ea504163a1e17561be9a5130 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:55.720273 14286 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
I20240501 20:01:55.720449 12448 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:55.790915 18295 consensus_queue.cc:579] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.362s)
I20240501 20:01:55.797525 12725 consensus_queue.cc:237] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:01:55.799358 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): Couldn't send request to peer 2d52575d9e1b40dca0fc553d7295eb74. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:55.799789 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.833788 18610 consensus_queue.cc:579] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.079s)
W20240501 20:01:55.836721 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:01:55.840368 14286 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:01:55.842074 12855 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:01:55.842555 18680 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:01:55.845499 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:55.846345 14222 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): Couldn't send request to peer 2d52575d9e1b40dca0fc553d7295eb74. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.846463 18610 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 20:01:55.848011 12855 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 20:01:55.849068 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet cf6948589837482a9f580731e3b861b8 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:55.849756 12452 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 10 to 11, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:55.852355 12793 connection.cc:535] client connection to 127.5.139.71:46083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:01:55.852437 12531 connection.cc:535] client connection to 127.5.139.71:46083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:01:55.852519 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 19 similar messages]
W20240501 20:01:55.852751 12924 connection.cc:535] client connection to 127.5.139.71:46083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:01:55.853027 12662 connection.cc:535] client connection to 127.5.139.71:46083 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:01:55.853283 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:55.853482 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:01:55.853559 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:55.853878 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:55.853976 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:55.854219 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:55.854286 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:55.859114 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.876915 9424 consensus_queue.cc:579] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.058s)
W20240501 20:01:55.879599 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.883270 14679 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 20:01:55.884724 14286 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:01:55.884827 12855 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:01:55.885198 18687 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:01:55.885391 18686 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:01:55.887233 9424 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:55.887758 12855 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:55.887764 14286 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:55.888796 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04ccc8addc55441b9ab9d8265b90b03b with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:55.890229 12448 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 11 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 2 leader_uuid: "c6d2c7096e404ea9a6f73b123d74c622" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:55.893688 14679 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:01:55.895267 14286 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 20:01:55.895591 18687 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:01:55.895740 12855 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 20:01:55.896310 18687 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:01:55.897264 18686 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 20:01:55.897698 14286 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 20:01:55.898665 12439 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 04ccc8addc55441b9ab9d8265b90b03b with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:55.899559 12855 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
W20240501 20:01:55.900130 14616 consensus_peers.cc:485] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997): Couldn't send request to peer 2d52575d9e1b40dca0fc553d7295eb74. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.900542 12453 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 13 to 14, NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) added. New cstate: current_term: 2 leader_uuid: "c6d2c7096e404ea9a6f73b123d74c622" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } } }
I20240501 20:01:55.913154 18696 ts_tablet_manager.cc:902] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 20:01:55.914057 18696 tablet_copy_client.cc:308] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 20:01:55.914891 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 851eeb0ab3364529907d9d09f2a4022e from peer a238fc0c6d844cedb8adccb97d01ae5e ({username='slave'} at 127.5.139.67:36493)
I20240501 20:01:55.915012 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 851eeb0ab3364529907d9d09f2a4022e from peer a238fc0c6d844cedb8adccb97d01ae5e at {username='slave'} at 127.5.139.67:36493: session id = a238fc0c6d844cedb8adccb97d01ae5e-851eeb0ab3364529907d9d09f2a4022e
I20240501 20:01:55.916538 13007 tablet_copy_source_session.cc:205] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:01:55.917575 18696 ts_tablet_manager.cc:1875] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:01:55.919561 18696 ts_tablet_manager.cc:1888] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20240501 20:01:55.919751 18696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 851eeb0ab3364529907d9d09f2a4022e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:55.920801 18696 tablet_copy_client.cc:783] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Starting download of 5 data blocks...
I20240501 20:01:55.926194 18696 tablet_copy_client.cc:647] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:55.927517 18696 tablet_copy_client.cc:523] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:55.928956 18696 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
W20240501 20:01:55.929913 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:55.939733 18696 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:55.940448 18696 tablet_bootstrap.cc:492] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap complete.
I20240501 20:01:55.940639 18696 ts_tablet_manager.cc:1367] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.006s
I20240501 20:01:55.941030 18696 raft_consensus.cc:348] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 20:01:55.941187 18696 raft_consensus.cc:729] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: LEARNER
I20240501 20:01:55.941283 18696 consensus_queue.cc:260] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 20:01:55.941546 18696 ts_tablet_manager.cc:1398] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:01:55.942154 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session a238fc0c6d844cedb8adccb97d01ae5e-851eeb0ab3364529907d9d09f2a4022e received from {username='slave'} at 127.5.139.67:36493
I20240501 20:01:55.942230 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session a238fc0c6d844cedb8adccb97d01ae5e-851eeb0ab3364529907d9d09f2a4022e on tablet 851eeb0ab3364529907d9d09f2a4022e with peer a238fc0c6d844cedb8adccb97d01ae5e
I20240501 20:01:55.978713 18734 ts_tablet_manager.cc:902] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 20:01:55.979501 18734 tablet_copy_client.cc:308] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 20:01:55.980131 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 706878734216445f8422e422228d4230 from peer ebbbd9ac07a94e8dbbae6def0a2df415 ({username='slave'} at 127.5.139.65:51979)
I20240501 20:01:55.980237 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 706878734216445f8422e422228d4230 from peer ebbbd9ac07a94e8dbbae6def0a2df415 at {username='slave'} at 127.5.139.65:51979: session id = ebbbd9ac07a94e8dbbae6def0a2df415-706878734216445f8422e422228d4230
I20240501 20:01:55.981295 13007 tablet_copy_source_session.cc:205] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:01:55.982559 18734 ts_tablet_manager.cc:1875] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:01:55.985146 18734 ts_tablet_manager.cc:1888] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.11
I20240501 20:01:55.985435 18734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 706878734216445f8422e422228d4230. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:55.987143 18734 tablet_copy_client.cc:783] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Starting download of 5 data blocks...
I20240501 20:01:55.990355 12967 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.997309 12704 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:01:55.997462 12836 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:55.997546 14660 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:55.998414 12574 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 20:01:56.005326 18734 tablet_copy_client.cc:647] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:56.013407 18734 tablet_copy_client.cc:523] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:56.016806 18734 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 20:01:56.019650 14266 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:56.038683 18734 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:56.039008 18734 tablet_bootstrap.cc:492] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap complete.
I20240501 20:01:56.039223 18734 ts_tablet_manager.cc:1367] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.000s
I20240501 20:01:56.039741 18734 raft_consensus.cc:348] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:01:56.039849 18734 raft_consensus.cc:729] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: LEARNER
I20240501 20:01:56.039963 18734 consensus_queue.cc:260] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:01:56.040272 18734 ts_tablet_manager.cc:1398] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:01:56.046663 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session ebbbd9ac07a94e8dbbae6def0a2df415-706878734216445f8422e422228d4230 received from {username='slave'} at 127.5.139.65:51979
I20240501 20:01:56.046753 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session ebbbd9ac07a94e8dbbae6def0a2df415-706878734216445f8422e422228d4230 on tablet 706878734216445f8422e422228d4230 with peer ebbbd9ac07a94e8dbbae6def0a2df415
W20240501 20:01:56.050937 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:56.064796 14729 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskuLQw07/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 19:56:48.160959 14737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:48.161212 14737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:48.165642 14737 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:49.178645 14737 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 19:56:49.709724 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.711316 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.711432 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.712114 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.712225 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.712334 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.712397 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.712460 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.712515 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.712569 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.712620 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.712713 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.713492 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.713582 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:49.757212 14737 rebalancer_tool.cc:1084] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move ignored: Illegal state: tablet 3e84a3865ce64eaebad4df46ec01842d: replica c6d2c7096e404ea9a6f73b123d74c622 is already present
I20240501 19:56:49.757369 14737 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:50.042886 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.043004 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.043107 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.043149 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.043184 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.043275 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.043354 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.043404 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.043458 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.043503 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
I20240501 19:56:50.098152 14737 rebalancer_tool.cc:1074] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:50.250303 14737 rebalancer_tool.cc:1074] tablet 851eeb0ab3364529907d9d09f2a4022e: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:50.414136 14737 rebalancer_tool.cc:1084] tablet 8bc9fe9ade8246779164d6db31b11e16: '2d52575d9e1b40dca0fc553d7295eb74' -> 'c6d2c7096e404ea9a6f73b123d74c622' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: NON_VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: true } }
I20240501 19:56:50.414224 14737 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:50.987314 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.987432 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.987512 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.987540 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.987576 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.987619 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:50.987679 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server 2ca06dad25a645bca71b09741798194a to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.987751 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.987798 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.987838 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.987895 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.987929 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.987964 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.988001 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:50.988040 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:50.988101 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:50.988142 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 9069ec6fcdb248479037f2d37e0e851d
I20240501 19:56:51.218600 14737 rebalancer_tool.cc:1074] tablet 0bf64d9f3ce94db28b032cdcd795a255: '2d52575d9e1b40dca0fc553d7295eb74' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:51.374552 14737 rebalancer_tool.cc:1074] tablet e4b15761177f4cffb7aa5e4d16e42ac7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:51.526917 14737 rebalancer_tool.cc:1074] tablet 8b61e94fc7e541c8aa84957560508a31: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '' move scheduled
I20240501 19:56:51.703379 14737 rebalancer_tool.cc:1074] tablet 7d7ea564ea504163a1e17561be9a5130: '2d52575d9e1b40dca0fc553d7295eb74' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:51.908099 14737 rebalancer_tool.cc:1074] tablet cf6948589837482a9f580731e3b861b8: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:52.149454 14737 rebalancer_tool.cc:1074] tablet 745e39ec23db4f90bb8b1e8b49a50244: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:52.339955 14737 rebalancer_tool.cc:1084] tablet 61bc22cdc6ab45529066fe6f4d14d4c7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move ignored: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } }.
Pending config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 19:56:52.340036 14737 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:52.937348 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:52.937503 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:52.937606 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:52.937676 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
I20240501 19:56:52.958940 14737 rebalancer_tool.cc:1084] tablet 955edb7c12fa4b32ac58827f131931e3: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '' move ignored: Not found: Peer with uuid a238fc0c6d844cedb8adccb97d01ae5e not found in consensus config
I20240501 19:56:52.959026 14737 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:53.625167 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:53.625355 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:53.625398 14737 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:53.702433 14737 rebalancer_tool.cc:1074] tablet 955edb7c12fa4b32ac58827f131931e3: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:53.776944 14737 rebalancer_tool.cc:1074] tablet 04ccc8addc55441b9ab9d8265b90b03b: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:53.922379 14737 rebalancer_tool.cc:1074] tablet 7e4d79f5e11b4923acb2ce543884b54a: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move scheduled
I20240501 19:56:54.221972 14737 rebalancer_tool.cc:1074] tablet db240ec5f8b2444fb6114ad814f3cdb7: '58f750e4d9574c9ebfd3d9e57eefd411' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:54.346824 14737 rebalancer_tool.cc:1074] tablet 544caa38514643d5b81cd6ed85e0bae7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '58f750e4d9574c9ebfd3d9e57eefd411' move scheduled
I20240501 19:56:54.638847 14737 rebalancer_tool.cc:1136] tablet 0bf64d9f3ce94db28b032cdcd795a255: '2d52575d9e1b40dca0fc553d7295eb74' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
I20240501 19:56:54.647974 14737 rebalancer_tool.cc:1136] tablet e4b15761177f4cffb7aa5e4d16e42ac7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 19:56:54.881767 14737 rebalancer_tool.cc:1136] tablet 8b61e94fc7e541c8aa84957560508a31: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '' move completed: OK
I20240501 19:56:54.892834 14737 rebalancer_tool.cc:1136] tablet 745e39ec23db4f90bb8b1e8b49a50244: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
I20240501 19:56:54.906862 14737 rebalancer_tool.cc:1136] tablet 7d7ea564ea504163a1e17561be9a5130: '2d52575d9e1b40dca0fc553d7295eb74' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:54.941677 14737 rebalancer_tool.cc:1074] tablet ca3285abe3654723a6922672391e9423: '58f750e4d9574c9ebfd3d9e57eefd411' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:55.028374 14737 rebalancer_tool.cc:1136] tablet ca3285abe3654723a6922672391e9423: '58f750e4d9574c9ebfd3d9e57eefd411' -> '2ca06dad25a645bca71b09741798194a' move completed: Incomplete: tablet ca3285abe3654723a6922672391e9423, TS 58f750e4d9574c9ebfd3d9e57eefd411 -> TS 2ca06dad25a645bca71b09741798194a move failed, target replica disappeared
I20240501 19:56:55.146857 14737 rebalancer_tool.cc:1136] tablet 851eeb0ab3364529907d9d09f2a4022e: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move completed: OK
I20240501 19:56:55.970983 14737 rebalancer_tool.cc:1136] tablet 7e4d79f5e11b4923acb2ce543884b54a: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move completed: OK
I20240501 19:56:55.981638 14737 rebalancer_tool.cc:1136] tablet cf6948589837482a9f580731e3b861b8: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move completed: OK
I20240501 19:56:56.002667 14737 rebalancer_tool.cc:1136] tablet db240ec5f8b2444fb6114ad814f3cdb7: '58f750e4d9574c9ebfd3d9e57eefd411' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:56.042385 14737 rebalancer_tool.cc:1136] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:56.047751 14737 rebalancer_tool.cc:1136] tablet 544caa38514643d5b81cd6ed85e0bae7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '58f750e4d9574c9ebfd3d9e57eefd411' move completed: OK
I20240501 19:56:56.911849 14737 rebalancer_tool.cc:1136] tablet 955edb7c12fa4b32ac58827f131931e3: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 20:01:55.918552 14737 rebalancer_tool.cc:1136] tablet 04ccc8addc55441b9ab9d8265b90b03b: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '2ca06dad25a645bca71b09741798194a' move completed: Incomplete: tablet 04ccc8addc55441b9ab9d8265b90b03b, TS a238fc0c6d844cedb8adccb97d01ae5e -> TS 2ca06dad25a645bca71b09741798194a move failed, target replica disappeared
Illegal state: tablet server 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): unacceptable health status UNAVAILABLE
W20240501 20:01:56.065829 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:56.105974 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:56.115962 18685 consensus_queue.cc:579] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.113s)
I20240501 20:01:56.118865 14286 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 20:01:56.121308 12855 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
W20240501 20:01:56.121886 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:56.122028 18685 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:01:56.124979 18169 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
W20240501 20:01:56.125732 14219 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): Couldn't send request to peer 58f750e4d9574c9ebfd3d9e57eefd411. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:56.125607 12855 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 20:01:56.126822 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3e84a3865ce64eaebad4df46ec01842d with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:56.127895 12453 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 13 to 14, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:56.150779 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(851eeb0ab3364529907d9d09f2a4022e): 4834 bytes on disk
I20240501 20:01:56.151260 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(851eeb0ab3364529907d9d09f2a4022e) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 20:01:56.214665 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:56.251892 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 04ccc8addc55441b9ab9d8265b90b03b: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 20:01:56.260331 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(706878734216445f8422e422228d4230): 4581 bytes on disk
I20240501 20:01:56.260784 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(706878734216445f8422e422228d4230) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4}
I20240501 20:01:56.271047 18696 ts_tablet_manager.cc:902] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 20:01:56.271585 18696 tablet_copy_client.cc:308] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 20:01:56.275533 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet 7d7ea564ea504163a1e17561be9a5130 from peer a238fc0c6d844cedb8adccb97d01ae5e ({username='slave'} at 127.5.139.67:60523)
I20240501 20:01:56.275633 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet 7d7ea564ea504163a1e17561be9a5130 from peer a238fc0c6d844cedb8adccb97d01ae5e at {username='slave'} at 127.5.139.67:60523: session id = a238fc0c6d844cedb8adccb97d01ae5e-7d7ea564ea504163a1e17561be9a5130
I20240501 20:01:56.276592 12614 tablet_copy_source_session.cc:205] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:01:56.277490 18696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d7ea564ea504163a1e17561be9a5130. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:56.279534 18696 tablet_copy_client.cc:783] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Starting download of 5 data blocks...
I20240501 20:01:56.281248 18776 ts_tablet_manager.cc:902] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 20:01:56.281848 18776 tablet_copy_client.cc:308] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 20:01:56.282337 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 12a50be843fb49ee8a66c9a17d271d61 from peer 2d52575d9e1b40dca0fc553d7295eb74 ({username='slave'} at 127.5.139.68:58755)
I20240501 20:01:56.282447 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 12a50be843fb49ee8a66c9a17d271d61 from peer 2d52575d9e1b40dca0fc553d7295eb74 at {username='slave'} at 127.5.139.68:58755: session id = 2d52575d9e1b40dca0fc553d7295eb74-12a50be843fb49ee8a66c9a17d271d61
I20240501 20:01:56.283807 12746 tablet_copy_source_session.cc:205] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:01:56.284235 18696 tablet_copy_client.cc:647] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:56.284677 18776 ts_tablet_manager.cc:1875] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:01:56.285501 18696 tablet_copy_client.cc:523] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:56.286571 18696 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 20:01:56.286772 18776 ts_tablet_manager.cc:1888] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 20:01:56.286950 18776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12a50be843fb49ee8a66c9a17d271d61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:56.287848 18776 tablet_copy_client.cc:783] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 5 data blocks...
I20240501 20:01:56.291955 18782 ts_tablet_manager.cc:902] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Initiating tablet copy from peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
I20240501 20:01:56.292392 18782 tablet_copy_client.cc:308] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.72:37453
I20240501 20:01:56.295292 18696 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:56.295790 18696 tablet_bootstrap.cc:492] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap complete.
I20240501 20:01:56.295955 18696 ts_tablet_manager.cc:1367] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.004s
I20240501 20:01:56.296326 18696 raft_consensus.cc:348] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 20:01:56.296458 18696 raft_consensus.cc:729] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: LEARNER
I20240501 20:01:56.296563 18696 consensus_queue.cc:260] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 20:01:56.296696 14700 tablet_copy_service.cc:140] P c6d2c7096e404ea9a6f73b123d74c622: Received BeginTabletCopySession request for tablet 04ccc8addc55441b9ab9d8265b90b03b from peer 2d52575d9e1b40dca0fc553d7295eb74 ({username='slave'} at 127.5.139.68:45901)
I20240501 20:01:56.296816 14700 tablet_copy_service.cc:161] P c6d2c7096e404ea9a6f73b123d74c622: Beginning new tablet copy session on tablet 04ccc8addc55441b9ab9d8265b90b03b from peer 2d52575d9e1b40dca0fc553d7295eb74 at {username='slave'} at 127.5.139.68:45901: session id = 2d52575d9e1b40dca0fc553d7295eb74-04ccc8addc55441b9ab9d8265b90b03b
I20240501 20:01:56.297408 18696 ts_tablet_manager.cc:1398] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:01:56.297834 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session a238fc0c6d844cedb8adccb97d01ae5e-7d7ea564ea504163a1e17561be9a5130 received from {username='slave'} at 127.5.139.67:60523
I20240501 20:01:56.297912 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session a238fc0c6d844cedb8adccb97d01ae5e-7d7ea564ea504163a1e17561be9a5130 on tablet 7d7ea564ea504163a1e17561be9a5130 with peer a238fc0c6d844cedb8adccb97d01ae5e
I20240501 20:01:56.298506 18776 tablet_copy_client.cc:647] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:56.298545 14700 tablet_copy_source_session.cc:205] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:01:56.299669 18782 ts_tablet_manager.cc:1875] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:01:56.299801 18776 tablet_copy_client.cc:523] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:56.301193 18776 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 20:01:56.301587 18782 ts_tablet_manager.cc:1888] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 20:01:56.301739 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ccc8addc55441b9ab9d8265b90b03b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:56.302665 18782 tablet_copy_client.cc:783] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 5 data blocks...
I20240501 20:01:56.306860 18782 tablet_copy_client.cc:647] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:56.308099 18782 tablet_copy_client.cc:523] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:56.309018 18776 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 20:01:56.309310 18782 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 20:01:56.309367 18776 tablet_bootstrap.cc:492] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap complete.
I20240501 20:01:56.309528 18776 ts_tablet_manager.cc:1367] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20240501 20:01:56.309906 18776 raft_consensus.cc:348] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 NON_PARTICIPANT]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 20:01:56.310129 18776 raft_consensus.cc:729] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: LEARNER
I20240501 20:01:56.310243 18776 consensus_queue.cc:260] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:01:56.310523 18776 ts_tablet_manager.cc:1398] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:01:56.311532 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-12a50be843fb49ee8a66c9a17d271d61 received from {username='slave'} at 127.5.139.68:58755
I20240501 20:01:56.311614 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-12a50be843fb49ee8a66c9a17d271d61 on tablet 12a50be843fb49ee8a66c9a17d271d61 with peer 2d52575d9e1b40dca0fc553d7295eb74
I20240501 20:01:56.319442 18782 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:56.319674 18782 tablet_bootstrap.cc:492] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap complete.
I20240501 20:01:56.319814 18782 ts_tablet_manager.cc:1367] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20240501 20:01:56.320137 18782 raft_consensus.cc:348] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:01:56.320256 18782 raft_consensus.cc:729] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: LEARNER
I20240501 20:01:56.320501 18782 consensus_queue.cc:260] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:01:56.320776 18782 ts_tablet_manager.cc:1398] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.003s sys 0.000s
I20240501 20:01:56.321671 14700 tablet_copy_service.cc:342] P c6d2c7096e404ea9a6f73b123d74c622: Request end of tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-04ccc8addc55441b9ab9d8265b90b03b received from {username='slave'} at 127.5.139.68:45901
I20240501 20:01:56.321775 14700 tablet_copy_service.cc:433] P c6d2c7096e404ea9a6f73b123d74c622: ending tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-04ccc8addc55441b9ab9d8265b90b03b on tablet 04ccc8addc55441b9ab9d8265b90b03b with peer 2d52575d9e1b40dca0fc553d7295eb74
I20240501 20:01:56.351649 12855 raft_consensus.cc:1203] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240501 20:01:56.365536 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(12a50be843fb49ee8a66c9a17d271d61): 4921 bytes on disk
I20240501 20:01:56.365955 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(12a50be843fb49ee8a66c9a17d271d61) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4}
I20240501 20:01:56.366613 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(04ccc8addc55441b9ab9d8265b90b03b): 4579 bytes on disk
I20240501 20:01:56.366925 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(04ccc8addc55441b9ab9d8265b90b03b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":40,"lbm_reads_lt_1ms":4}
I20240501 20:01:56.402769 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(7d7ea564ea504163a1e17561be9a5130): 4834 bytes on disk
I20240501 20:01:56.403219 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(7d7ea564ea504163a1e17561be9a5130) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 20:01:56.441869 18782 ts_tablet_manager.cc:902] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Initiating tablet copy from peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719)
I20240501 20:01:56.442389 18782 tablet_copy_client.cc:308] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.69:34719
I20240501 20:01:56.442976 14306 tablet_copy_service.cc:140] P 9069ec6fcdb248479037f2d37e0e851d: Received BeginTabletCopySession request for tablet cf6948589837482a9f580731e3b861b8 from peer 2d52575d9e1b40dca0fc553d7295eb74 ({username='slave'} at 127.5.139.68:47587)
I20240501 20:01:56.443086 14306 tablet_copy_service.cc:161] P 9069ec6fcdb248479037f2d37e0e851d: Beginning new tablet copy session on tablet cf6948589837482a9f580731e3b861b8 from peer 2d52575d9e1b40dca0fc553d7295eb74 at {username='slave'} at 127.5.139.68:47587: session id = 2d52575d9e1b40dca0fc553d7295eb74-cf6948589837482a9f580731e3b861b8
I20240501 20:01:56.444482 14306 tablet_copy_source_session.cc:205] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:01:56.445513 18782 ts_tablet_manager.cc:1875] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:01:56.447470 18782 ts_tablet_manager.cc:1888] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 20:01:56.447647 18782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf6948589837482a9f580731e3b861b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:56.448534 18782 tablet_copy_client.cc:783] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 5 data blocks...
I20240501 20:01:56.453178 18782 tablet_copy_client.cc:647] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:56.454524 18782 tablet_copy_client.cc:523] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:56.455590 18782 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap starting.
I20240501 20:01:56.464422 18782 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:56.464697 18782 tablet_bootstrap.cc:492] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Bootstrap complete.
I20240501 20:01:56.465003 18782 ts_tablet_manager.cc:1367] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent bootstrapping tablet: real 0.009s user 0.010s sys 0.000s
I20240501 20:01:56.465489 18782 raft_consensus.cc:348] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:01:56.465610 18782 raft_consensus.cc:729] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Initialized, Role: LEARNER
I20240501 20:01:56.465715 18782 consensus_queue.cc:260] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:01:56.466028 18782 ts_tablet_manager.cc:1398] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:01:56.466626 14306 tablet_copy_service.cc:342] P 9069ec6fcdb248479037f2d37e0e851d: Request end of tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-cf6948589837482a9f580731e3b861b8 received from {username='slave'} at 127.5.139.68:47587
I20240501 20:01:56.466737 14306 tablet_copy_service.cc:433] P 9069ec6fcdb248479037f2d37e0e851d: ending tablet copy session 2d52575d9e1b40dca0fc553d7295eb74-cf6948589837482a9f580731e3b861b8 on tablet cf6948589837482a9f580731e3b861b8 with peer 2d52575d9e1b40dca0fc553d7295eb74
I20240501 20:01:56.494014 12595 raft_consensus.cc:1203] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240501 20:01:56.509809 18806 ts_tablet_manager.cc:902] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Initiating tablet copy from peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719)
I20240501 20:01:56.510339 18806 tablet_copy_client.cc:308] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.69:34719
I20240501 20:01:56.510810 14306 tablet_copy_service.cc:140] P 9069ec6fcdb248479037f2d37e0e851d: Received BeginTabletCopySession request for tablet 3e84a3865ce64eaebad4df46ec01842d from peer 58f750e4d9574c9ebfd3d9e57eefd411 ({username='slave'} at 127.5.139.66:58123)
I20240501 20:01:56.510918 14306 tablet_copy_service.cc:161] P 9069ec6fcdb248479037f2d37e0e851d: Beginning new tablet copy session on tablet 3e84a3865ce64eaebad4df46ec01842d from peer 58f750e4d9574c9ebfd3d9e57eefd411 at {username='slave'} at 127.5.139.66:58123: session id = 58f750e4d9574c9ebfd3d9e57eefd411-3e84a3865ce64eaebad4df46ec01842d
I20240501 20:01:56.511966 14306 tablet_copy_source_session.cc:205] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:01:56.512681 18806 ts_tablet_manager.cc:1875] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:01:56.514736 18806 ts_tablet_manager.cc:1888] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.12
I20240501 20:01:56.514952 18806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e84a3865ce64eaebad4df46ec01842d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 20:01:56.516026 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:56.516201 18806 tablet_copy_client.cc:783] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 5 data blocks...
I20240501 20:01:56.520464 18806 tablet_copy_client.cc:647] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:56.521632 18806 tablet_copy_client.cc:523] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:56.522739 18806 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 20:01:56.531852 18806 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:56.532167 18806 tablet_bootstrap.cc:492] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap complete.
I20240501 20:01:56.532392 18806 ts_tablet_manager.cc:1367] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.010s user 0.011s sys 0.000s
I20240501 20:01:56.532811 18806 raft_consensus.cc:348] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 20:01:56.532945 18806 raft_consensus.cc:729] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: LEARNER
I20240501 20:01:56.533080 18806 consensus_queue.cc:260] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 20:01:56.533522 18806 ts_tablet_manager.cc:1398] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:01:56.534133 14306 tablet_copy_service.cc:342] P 9069ec6fcdb248479037f2d37e0e851d: Request end of tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-3e84a3865ce64eaebad4df46ec01842d received from {username='slave'} at 127.5.139.66:58123
I20240501 20:01:56.534216 14306 tablet_copy_service.cc:433] P 9069ec6fcdb248479037f2d37e0e851d: ending tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-3e84a3865ce64eaebad4df46ec01842d on tablet 3e84a3865ce64eaebad4df46ec01842d with peer 58f750e4d9574c9ebfd3d9e57eefd411
W20240501 20:01:56.588219 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:01:56.601580 18787 consensus_queue.cc:579] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.123s)
W20240501 20:01:56.602883 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:01:56.607208 12855 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 20:01:56.608587 14286 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:01:56.608636 14679 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:01:56.609192 18787 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:01:56.609618 18787 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:01:56.611282 18043 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:56.611660 14679 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:56.613524 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 557994658e2c4c8bbc1eeacdab958ddd with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:56.614418 14286 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:01:56.614452 12453 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 11 to 12, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:56.618438 13035 maintenance_manager.cc:419] P 2d52575d9e1b40dca0fc553d7295eb74: Scheduling UndoDeltaBlockGCOp(cf6948589837482a9f580731e3b861b8): 4834 bytes on disk
I20240501 20:01:56.618863 12925 maintenance_manager.cc:643] P 2d52575d9e1b40dca0fc553d7295eb74: UndoDeltaBlockGCOp(cf6948589837482a9f580731e3b861b8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 20:01:56.619050 12855 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:01:56.620776 14679 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:01:56.621258 18822 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 20:01:56.621542 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): Couldn't send request to peer ebbbd9ac07a94e8dbbae6def0a2df415. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:56.621888 14286 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 20:01:56.622211 18822 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:01:56.623261 18043 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } } }
I20240501 20:01:56.623694 14679 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } } }
I20240501 20:01:56.624076 14286 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } } }
I20240501 20:01:56.624718 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 557994658e2c4c8bbc1eeacdab958ddd with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:56.625716 12453 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 12 to 13, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:01:56.703187 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:01:56.727835 12988 raft_consensus.cc:1203] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
I20240501 20:01:56.736146 12855 raft_consensus.cc:1203] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 20:01:56.752856 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(3e84a3865ce64eaebad4df46ec01842d): 4921 bytes on disk
I20240501 20:01:56.753361 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(3e84a3865ce64eaebad4df46ec01842d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 20:01:56.772810 12988 raft_consensus.cc:1203] T 12a50be843fb49ee8a66c9a17d271d61 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 20:01:56.789425 18763 raft_consensus.cc:1050] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e to VOTER
I20240501 20:01:56.789986 18763 consensus_queue.cc:237] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } }
I20240501 20:01:56.791349 12855 raft_consensus.cc:1261] T 851eeb0ab3364529907d9d09f2a4022e P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:01:56.791749 18834 consensus_queue.cc:1035] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 20:01:56.791960 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:56.792276 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:56.799834 14221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 22 similar messages]
I20240501 20:01:56.905421 18687 raft_consensus.cc:482] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1d49e80be3d04fb99b711c62d61bac09)
I20240501 20:01:56.905575 18687 raft_consensus.cc:504] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 20:01:56.905977 18687 leader_election.cc:288] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997), 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 20:01:56.906386 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "c6d2c7096e404ea9a6f73b123d74c622" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
W20240501 20:01:56.907148 14616 leader_election.cc:334] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:56.907223 14616 leader_election.cc:302] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6d2c7096e404ea9a6f73b123d74c622; no voters: 1d49e80be3d04fb99b711c62d61bac09, 2d52575d9e1b40dca0fc553d7295eb74
I20240501 20:01:56.907358 18687 raft_consensus.cc:2726] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:01:56.995352 18612 raft_consensus.cc:1050] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 to VOTER
I20240501 20:01:56.995786 18612 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } }
I20240501 20:01:56.997153 12595 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:01:56.997197 14286 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:01:56.997880 18764 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:01:56.998142 18612 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 20:01:56.998317 12923 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:56.999869 18612 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:57.000319 14286 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:57.002231 12595 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:57.002676 12444 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 13 to 14, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:57.005887 12988 consensus_queue.cc:237] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } }
W20240501 20:01:57.005995 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:57.006968 14286 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 20:01:57.007023 12595 raft_consensus.cc:1261] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 20:01:57.007319 18763 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:01:57.007428 18834 consensus_queue.cc:1035] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:01:57.008765 18612 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:57.009209 14286 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:57.010233 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 706878734216445f8422e422228d4230 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:57.010732 12595 raft_consensus.cc:2932] T 706878734216445f8422e422228d4230 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:57.011222 12453 catalog_manager.cc:5484] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 14 to 15, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:57.033229 12988 raft_consensus.cc:1203] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20240501 20:01:57.050828 12725 raft_consensus.cc:1203] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
I20240501 20:01:57.117414 18857 ts_tablet_manager.cc:902] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 20:01:57.117874 18857 tablet_copy_client.cc:308] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 20:01:57.118311 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 557994658e2c4c8bbc1eeacdab958ddd from peer ebbbd9ac07a94e8dbbae6def0a2df415 ({username='slave'} at 127.5.139.65:35883)
I20240501 20:01:57.118407 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 557994658e2c4c8bbc1eeacdab958ddd from peer ebbbd9ac07a94e8dbbae6def0a2df415 at {username='slave'} at 127.5.139.65:35883: session id = ebbbd9ac07a94e8dbbae6def0a2df415-557994658e2c4c8bbc1eeacdab958ddd
I20240501 20:01:57.119429 12877 tablet_copy_source_session.cc:205] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:01:57.120116 18857 ts_tablet_manager.cc:1875] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:01:57.122071 18857 ts_tablet_manager.cc:1888] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 20:01:57.122246 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 557994658e2c4c8bbc1eeacdab958ddd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:57.123294 18857 tablet_copy_client.cc:783] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Starting download of 5 data blocks...
I20240501 20:01:57.125089 18834 raft_consensus.cc:482] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 1d49e80be3d04fb99b711c62d61bac09)
I20240501 20:01:57.125213 18834 raft_consensus.cc:504] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 20:01:57.125742 18834 leader_election.cc:288] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453), 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 20:01:57.126183 14679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "c6d2c7096e404ea9a6f73b123d74c622" is_pre_election: true
I20240501 20:01:57.126330 14679 raft_consensus.cc:2454] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 2.
I20240501 20:01:57.126663 12923 leader_election.cc:302] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, c6d2c7096e404ea9a6f73b123d74c622; no voters:
I20240501 20:01:57.126824 18834 raft_consensus.cc:2781] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:01:57.126876 18834 raft_consensus.cc:482] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 1d49e80be3d04fb99b711c62d61bac09)
I20240501 20:01:57.126915 18834 raft_consensus.cc:3037] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:57.127326 12924 leader_election.cc:334] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:57.127862 18834 raft_consensus.cc:504] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 20:01:57.128147 18834 leader_election.cc:288] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 election: Requested vote from peers c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453), 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083)
I20240501 20:01:57.128422 14679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "955edb7c12fa4b32ac58827f131931e3" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "c6d2c7096e404ea9a6f73b123d74c622"
I20240501 20:01:57.128556 14679 raft_consensus.cc:3037] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:01:57.129002 12924 leader_election.cc:334] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:57.129663 14679 raft_consensus.cc:2454] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52575d9e1b40dca0fc553d7295eb74 in term 3.
I20240501 20:01:57.130199 12923 leader_election.cc:302] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, c6d2c7096e404ea9a6f73b123d74c622; no voters: 1d49e80be3d04fb99b711c62d61bac09
I20240501 20:01:57.130414 18834 raft_consensus.cc:2781] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:01:57.130508 18834 raft_consensus.cc:686] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [term 3 LEADER]: Becoming Leader. State: Replica: 2d52575d9e1b40dca0fc553d7295eb74, State: Running, Role: LEADER
I20240501 20:01:57.130661 18857 tablet_copy_client.cc:647] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:57.130640 18834 consensus_queue.cc:237] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }
I20240501 20:01:57.132145 18857 tablet_copy_client.cc:523] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:57.132474 12444 catalog_manager.cc:5484] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: term changed from 2 to 3, leader changed from 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) to 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68). New cstate: current_term: 3 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:57.133239 18857 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 20:01:57.141453 18857 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:57.141721 18857 tablet_bootstrap.cc:492] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap complete.
I20240501 20:01:57.141868 18857 ts_tablet_manager.cc:1367] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20240501 20:01:57.142197 18857 raft_consensus.cc:348] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:01:57.142294 18857 raft_consensus.cc:729] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: LEARNER
I20240501 20:01:57.142385 18857 consensus_queue.cc:260] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:01:57.142666 18857 ts_tablet_manager.cc:1398] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:01:57.143095 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session ebbbd9ac07a94e8dbbae6def0a2df415-557994658e2c4c8bbc1eeacdab958ddd received from {username='slave'} at 127.5.139.65:35883
I20240501 20:01:57.143162 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session ebbbd9ac07a94e8dbbae6def0a2df415-557994658e2c4c8bbc1eeacdab958ddd on tablet 557994658e2c4c8bbc1eeacdab958ddd with peer ebbbd9ac07a94e8dbbae6def0a2df415
I20240501 20:01:57.175328 18687 consensus_queue.cc:579] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.414s)
I20240501 20:01:57.191144 18687 raft_consensus.cc:1050] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622: attempting to promote NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 to VOTER
I20240501 20:01:57.191586 18687 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:01:57.192770 12988 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 20:01:57.192935 14286 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 20:01:57.193023 12855 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 20:01:57.193375 18817 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:01:57.193547 18687 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:01:57.193655 18870 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:01:57.195601 18870 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.196030 12988 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.196236 14286 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.197422 12855 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.198864 12444 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 14 to 15, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c6d2c7096e404ea9a6f73b123d74c622" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:57.201732 18834 consensus_queue.cc:579] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.012s)
I20240501 20:01:57.202432 14679 consensus_queue.cc:237] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:01:57.203891 12988 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240501 20:01:57.204123 14286 raft_consensus.cc:1261] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Refusing update from remote peer c6d2c7096e404ea9a6f73b123d74c622: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240501 20:01:57.204716 18818 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 20:01:57.205030 18818 consensus_queue.cc:1035] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 20:01:57.206967 18818 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.207165 12988 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:01:57.207401 12986 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.209090 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04ccc8addc55441b9ab9d8265b90b03b with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 20:01:57.209520 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:57.209487 12453 catalog_manager.cc:5484] T 04ccc8addc55441b9ab9d8265b90b03b P c6d2c7096e404ea9a6f73b123d74c622 reported cstate change: config changed from index 15 to 16, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New cstate: current_term: 2 leader_uuid: "c6d2c7096e404ea9a6f73b123d74c622" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:57.209906 14286 raft_consensus.cc:2932] T 04ccc8addc55441b9ab9d8265b90b03b P 9069ec6fcdb248479037f2d37e0e851d [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.211644 12725 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:01:57.212111 18764 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:01:57.212467 12836 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ccc8addc55441b9ab9d8265b90b03b with delete_type TABLET_DATA_TOMBSTONED (TS a238fc0c6d844cedb8adccb97d01ae5e not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:45158
I20240501 20:01:57.213253 18875 tablet_replica.cc:330] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: stopping tablet replica
I20240501 20:01:57.213403 18875 raft_consensus.cc:2229] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:57.213560 18875 raft_consensus.cc:2258] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:57.214416 18875 ts_tablet_manager.cc:1875] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:57.214610 18834 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.215006 12725 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 11 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.216012 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8bc9fe9ade8246779164d6db31b11e16 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:57.217305 12450 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 11 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.217808 18875 ts_tablet_manager.cc:1888] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.15
I20240501 20:01:57.217912 18875 log.cc:1196] T 04ccc8addc55441b9ab9d8265b90b03b P a238fc0c6d844cedb8adccb97d01ae5e: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-2/wal/wals/04ccc8addc55441b9ab9d8265b90b03b
I20240501 20:01:57.218541 12437 catalog_manager.cc:4830] TS a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): tablet 04ccc8addc55441b9ab9d8265b90b03b (table rebalance_test_table_9 [id=b4c7884870794c24aee087d8ea5702d0]) successfully deleted
I20240501 20:01:57.220463 12988 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:01:57.221613 12725 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
W20240501 20:01:57.221803 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): Couldn't send request to peer ebbbd9ac07a94e8dbbae6def0a2df415. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:57.222127 18834 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 20:01:57.222321 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:57.223654 18764 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } } }
I20240501 20:01:57.224102 12725 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 13 to 14, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } } }
I20240501 20:01:57.225508 12440 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8bc9fe9ade8246779164d6db31b11e16 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:01:57.226308 12447 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 13 to 14, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:57.255467 18775 raft_consensus.cc:1050] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e to VOTER
I20240501 20:01:57.255856 18775 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } }
I20240501 20:01:57.257009 12855 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:01:57.257364 14286 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:01:57.257445 18882 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:01:57.257822 12530 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:57.257983 18882 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:01:57.259615 18775 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 20:01:57.260066 14286 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 20:01:57.261452 12855 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
W20240501 20:01:57.262537 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 20:01:57.262857 12450 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 10 to 11, a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:57.263849 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(557994658e2c4c8bbc1eeacdab958ddd): 4921 bytes on disk
I20240501 20:01:57.264273 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(557994658e2c4c8bbc1eeacdab958ddd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 20:01:57.266170 12595 consensus_queue.cc:237] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } }
I20240501 20:01:57.267694 12855 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:01:57.268114 18885 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:01:57.268594 14286 raft_consensus.cc:1261] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 20:01:57.269205 18885 consensus_queue.cc:1035] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:01:57.269608 18882 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 20:01:57.270107 12855 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 20:01:57.270980 14286 raft_consensus.cc:2932] T 7d7ea564ea504163a1e17561be9a5130 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } } }
I20240501 20:01:57.271232 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7d7ea564ea504163a1e17561be9a5130 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:57.272140 12447 catalog_manager.cc:5484] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 11 to 12, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:57.337549 18893 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:01:57.430752 18850 raft_consensus.cc:1050] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d: attempting to promote NON_VOTER 2d52575d9e1b40dca0fc553d7295eb74 to VOTER
I20240501 20:01:57.431177 18850 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:01:57.432605 12855 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:01:57.432991 12988 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEARNER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:01:57.433063 18900 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 20:01:57.433364 14221 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:57.433372 18900 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:01:57.435153 18851 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.435635 12988 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.436918 12855 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.438359 12453 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 11 to 12, 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.442622 14286 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:01:57.443887 12855 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:01:57.443965 12988 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 20:01:57.444316 18685 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:01:57.444463 18900 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:01:57.446112 18900 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.446764 12855 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.446816 12988 raft_consensus.cc:2932] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } }
I20240501 20:01:57.447726 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cf6948589837482a9f580731e3b861b8 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:57.449476 12450 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 12 to 13, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:57.484664 18850 raft_consensus.cc:1050] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d: attempting to promote NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 to VOTER
I20240501 20:01:57.485184 18850 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 10, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:01:57.487067 12725 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEARNER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 20:01:57.487579 18851 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 20:01:57.487742 14221 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:57.493306 12855 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 20:01:57.493870 18851 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:01:57.495908 18900 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:01:57.496533 12855 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:01:57.497997 12725 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:01:57.500454 12447 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 14 to 15, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:57.503986 14286 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:01:57.505227 12725 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240501 20:01:57.505653 18900 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 20:01:57.507043 12857 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 9069ec6fcdb248479037f2d37e0e851d: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240501 20:01:57.507861 18851 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 20:01:57.508141 18851 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:01:57.508867 12855 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:01:57.510057 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3e84a3865ce64eaebad4df46ec01842d with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:57.510087 12725 raft_consensus.cc:2932] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:01:57.512316 12447 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P 9069ec6fcdb248479037f2d37e0e851d reported cstate change: config changed from index 15 to 16, VOTER 2ca06dad25a645bca71b09741798194a (127.5.139.70) evicted. New cstate: current_term: 2 leader_uuid: "9069ec6fcdb248479037f2d37e0e851d" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:01:57.554658 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:01:57.570405 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:57.602279 18857 ts_tablet_manager.cc:902] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 20:01:57.602900 18857 tablet_copy_client.cc:308] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 20:01:57.603437 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet 8bc9fe9ade8246779164d6db31b11e16 from peer ebbbd9ac07a94e8dbbae6def0a2df415 ({username='slave'} at 127.5.139.65:51979)
I20240501 20:01:57.603554 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet 8bc9fe9ade8246779164d6db31b11e16 from peer ebbbd9ac07a94e8dbbae6def0a2df415 at {username='slave'} at 127.5.139.65:51979: session id = ebbbd9ac07a94e8dbbae6def0a2df415-8bc9fe9ade8246779164d6db31b11e16
I20240501 20:01:57.604555 13007 tablet_copy_source_session.cc:205] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:01:57.605304 18857 ts_tablet_manager.cc:1875] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:01:57.607172 18857 ts_tablet_manager.cc:1888] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 20:01:57.607331 18857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bc9fe9ade8246779164d6db31b11e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:01:57.608234 18857 tablet_copy_client.cc:783] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Starting download of 5 data blocks...
I20240501 20:01:57.612839 18857 tablet_copy_client.cc:647] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Starting download of 1 WAL segments...
I20240501 20:01:57.614141 18857 tablet_copy_client.cc:523] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:01:57.615272 18857 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 20:01:57.620028 14680 raft_consensus.cc:1261] T 955edb7c12fa4b32ac58827f131931e3 P c6d2c7096e404ea9a6f73b123d74c622 [term 3 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20240501 20:01:57.620481 18764 consensus_queue.cc:1035] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 20:01:57.626958 18857 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:01:57.627210 18857 tablet_bootstrap.cc:492] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap complete.
I20240501 20:01:57.627351 18857 ts_tablet_manager.cc:1367] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 20:01:57.627754 18857 raft_consensus.cc:348] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:01:57.627882 18857 raft_consensus.cc:729] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: LEARNER
I20240501 20:01:57.627991 18857 consensus_queue.cc:260] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:01:57.628361 18857 ts_tablet_manager.cc:1398] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240501 20:01:57.628840 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session ebbbd9ac07a94e8dbbae6def0a2df415-8bc9fe9ade8246779164d6db31b11e16 received from {username='slave'} at 127.5.139.65:51979
I20240501 20:01:57.628923 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session ebbbd9ac07a94e8dbbae6def0a2df415-8bc9fe9ade8246779164d6db31b11e16 on tablet 8bc9fe9ade8246779164d6db31b11e16 with peer ebbbd9ac07a94e8dbbae6def0a2df415
I20240501 20:01:57.718809 12595 raft_consensus.cc:1203] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240501 20:01:57.766435 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(8bc9fe9ade8246779164d6db31b11e16): 4921 bytes on disk
I20240501 20:01:57.766901 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(8bc9fe9ade8246779164d6db31b11e16) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240501 20:01:57.946702 18893 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:01:58.114513 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:58.163528 12595 raft_consensus.cc:1203] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.13->[2.14-2.14] Dedup: 2.14->[]
W20240501 20:01:58.167204 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:58.215961 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:58.222265 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:01:58.264055 18924 raft_consensus.cc:1050] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 to VOTER
I20240501 20:01:58.264480 18924 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } }
I20240501 20:01:58.265965 12595 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:01:58.266285 14679 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:01:58.266425 14286 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 20:01:58.266418 18960 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:01:58.266728 18924 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:01:58.266934 18961 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 20:01:58.268887 18924 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.269346 14286 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.271049 12595 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.271557 12450 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 13 to 14, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:01:58.273186 14679 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.278313 12855 consensus_queue.cc:237] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } }
I20240501 20:01:58.279336 12595 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 20:01:58.279430 14679 raft_consensus.cc:1261] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 20:01:58.279740 18961 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:01:58.279929 18960 consensus_queue.cc:1035] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:01:58.281100 12442 maintenance_manager.cc:419] P be1305e487d64b748456a721443e3b3b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 11860 bytes on disk
I20240501 20:01:58.281813 18924 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.282332 12595 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.283474 12441 maintenance_manager.cc:643] P be1305e487d64b748456a721443e3b3b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"thread_start_us":59,"threads_started":1}
I20240501 20:01:58.283663 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 557994658e2c4c8bbc1eeacdab958ddd with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:01:58.285090 14679 raft_consensus.cc:2932] T 557994658e2c4c8bbc1eeacdab958ddd P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.285157 12453 catalog_manager.cc:5484] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 14 to 15, VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:01:58.287766 14266 tablet_service.cc:1514] Processing DeleteTablet for tablet 557994658e2c4c8bbc1eeacdab958ddd with delete_type TABLET_DATA_TOMBSTONED (TS 9069ec6fcdb248479037f2d37e0e851d not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:57092
I20240501 20:01:58.288101 18970 tablet_replica.cc:330] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: stopping tablet replica
I20240501 20:01:58.288223 18970 raft_consensus.cc:2229] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 20:01:58.288460 18970 raft_consensus.cc:2258] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 20:01:58.289283 18970 ts_tablet_manager.cc:1875] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 20:01:58.291872 18970 ts_tablet_manager.cc:1888] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.14
I20240501 20:01:58.291939 18970 log.cc:1196] T 557994658e2c4c8bbc1eeacdab958ddd P 9069ec6fcdb248479037f2d37e0e851d: Deleting WAL directory at /tmp/dist-test-taskuLQw07/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714593358198074-5677-0/raft_consensus-itest-cluster/ts-4/wal/wals/557994658e2c4c8bbc1eeacdab958ddd
I20240501 20:01:58.292496 12437 catalog_manager.cc:4830] TS 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): tablet 557994658e2c4c8bbc1eeacdab958ddd (table rebalance_test_table_6 [id=fdfb935a205e456898e1e789d13f897d]) successfully deleted
W20240501 20:01:58.354526 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:58.364498 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:58.389355 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 20:01:58.422896 14660 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.423246 12967 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 20:01:58.425482 12704 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 20:01:58.427587 12836 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 20:01:58.431771 14266 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 20:01:58.439383 12574 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
W20240501 20:01:58.463532 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:58.464082 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:01:58.478574 14730 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskuLQw07/build/debug/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 19:56:48.225847 14736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 19:56:48.226125 14736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 19:56:48.230388 14736 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 19:56:49.223685 14736 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 19:56:49.605463 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.606194 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.606413 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.606462 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.606510 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.606550 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.606592 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.608734 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.608798 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.608836 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:49.608873 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.608914 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:49.608953 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:49.608994 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:49.609031 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:49.746590 14736 rebalancer_tool.cc:1074] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:49.801064 14736 rebalancer_tool.cc:1074] tablet 8b61e94fc7e541c8aa84957560508a31: '2d52575d9e1b40dca0fc553d7295eb74' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:49.846962 14736 rebalancer_tool.cc:1074] tablet 745e39ec23db4f90bb8b1e8b49a50244: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:49.870419 14736 rebalancer_tool.cc:1074] tablet 706878734216445f8422e422228d4230: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:49.926555 14736 rebalancer_tool.cc:1074] tablet 22b272a8582d4060a6a527ac4fce9ae5: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:49.982715 14736 rebalancer_tool.cc:1074] tablet 04ccc8addc55441b9ab9d8265b90b03b: '2d52575d9e1b40dca0fc553d7295eb74' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:50.037348 14736 rebalancer_tool.cc:1074] tablet 04a70a2ee2144f2287795ffe723d2716: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:50.081522 14736 rebalancer_tool.cc:1074] tablet db240ec5f8b2444fb6114ad814f3cdb7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:50.375838 14736 rebalancer_tool.cc:1074] tablet 7d7ea564ea504163a1e17561be9a5130: '58f750e4d9574c9ebfd3d9e57eefd411' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:50.489665 14736 rebalancer_tool.cc:1074] tablet cf6948589837482a9f580731e3b861b8: '2d52575d9e1b40dca0fc553d7295eb74' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:50.569449 14736 rebalancer_tool.cc:1084] tablet f08535cb2f46464b9e510a45fee67b4f: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move ignored: Illegal state: Request specified cas_config_opid_index of 4 but the committed config has opid_index of 5
I20240501 19:56:50.569538 14736 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:51.148002 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:51.148077 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:51.148154 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:51.148201 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:51.148276 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:51.148322 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:51.148363 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table fdfb935a205e456898e1e789d13f897d: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:51.148409 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:51.148512 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:51.148555 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:51.148604 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
I20240501 19:56:51.337998 14736 rebalancer_tool.cc:1074] tablet 7e4d79f5e11b4923acb2ce543884b54a: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:51.424697 14736 rebalancer_tool.cc:1074] tablet f08535cb2f46464b9e510a45fee67b4f: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:51.626938 14736 rebalancer_tool.cc:1074] tablet ce6add2f19a649b3a12497951047705c: '2d52575d9e1b40dca0fc553d7295eb74' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:51.681932 14736 rebalancer_tool.cc:1074] tablet 0bf64d9f3ce94db28b032cdcd795a255: '58f750e4d9574c9ebfd3d9e57eefd411' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:51.702482 14736 rebalancer_tool.cc:1084] tablet 0fb481d394034a8f85af613beb5c9ec8: '2ca06dad25a645bca71b09741798194a' -> '9069ec6fcdb248479037f2d37e0e851d' move ignored: Not found: Peer with uuid 2ca06dad25a645bca71b09741798194a not found in consensus config
I20240501 19:56:51.702577 14736 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:52.321251 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:52.321401 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:52.321516 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:52.321645 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:52.321708 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:52.321765 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 1d49e80be3d04fb99b711c62d61bac09
W20240501 19:56:52.321816 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 5f1f9662c1cf49858a630b67ffed42c4: could not find any suitable replica to move from server ebbbd9ac07a94e8dbbae6def0a2df415 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:52.321863 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:52.321954 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:52.353224 14736 rebalancer_tool.cc:1074] tablet 955edb7c12fa4b32ac58827f131931e3: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:52.451804 14736 rebalancer_tool.cc:1074] tablet 557994658e2c4c8bbc1eeacdab958ddd: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '9069ec6fcdb248479037f2d37e0e851d' move scheduled
I20240501 19:56:52.494518 14736 rebalancer_tool.cc:1074] tablet ca3285abe3654723a6922672391e9423: '58f750e4d9574c9ebfd3d9e57eefd411' -> '1d49e80be3d04fb99b711c62d61bac09' move scheduled
I20240501 19:56:52.516287 14736 rebalancer_tool.cc:1084] tablet 8bc9fe9ade8246779164d6db31b11e16: '2d52575d9e1b40dca0fc553d7295eb74' -> '9069ec6fcdb248479037f2d37e0e851d' move ignored: Not found: Peer with uuid 2d52575d9e1b40dca0fc553d7295eb74 not found in consensus config
I20240501 19:56:52.516361 14736 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:53.284243 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table f9f43040d1d7463b810225e4a7bce24c: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 9069ec6fcdb248479037f2d37e0e851d
W20240501 19:56:53.284330 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 2b92d6f20cdd41a4ac93f9cda49833c7: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:53.284416 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table c5dde3ea07b74da4b047a58bf0339e0d: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server 2ca06dad25a645bca71b09741798194a
W20240501 19:56:53.284695 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 6f10e15b627248ecb203396d1403dcdb: could not find any suitable replica to move from server 58f750e4d9574c9ebfd3d9e57eefd411 to server c6d2c7096e404ea9a6f73b123d74c622
W20240501 19:56:53.284767 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table e77f8c1c9f524b539586b4509dd26f73: could not find any suitable replica to move from server a238fc0c6d844cedb8adccb97d01ae5e to server c6d2c7096e404ea9a6f73b123d74c622
I20240501 19:56:53.412923 14736 rebalancer_tool.cc:1074] tablet ae2885f7933f462c96db9a99a4eb2090: '2d52575d9e1b40dca0fc553d7295eb74' -> '2ca06dad25a645bca71b09741798194a' move scheduled
I20240501 19:56:53.671739 14736 rebalancer_tool.cc:1136] tablet 8b61e94fc7e541c8aa84957560508a31: '2d52575d9e1b40dca0fc553d7295eb74' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 19:56:53.696341 14736 rebalancer_tool.cc:1136] tablet 04ccc8addc55441b9ab9d8265b90b03b: '2d52575d9e1b40dca0fc553d7295eb74' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:53.717958 14736 rebalancer_tool.cc:1136] tablet db240ec5f8b2444fb6114ad814f3cdb7: 'a238fc0c6d844cedb8adccb97d01ae5e' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
I20240501 19:56:53.768873 14736 rebalancer_tool.cc:1136] tablet 22b272a8582d4060a6a527ac4fce9ae5: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
I20240501 19:56:53.902814 14736 rebalancer_tool.cc:1136] tablet 7d7ea564ea504163a1e17561be9a5130: '58f750e4d9574c9ebfd3d9e57eefd411' -> '2ca06dad25a645bca71b09741798194a' move completed: OK
I20240501 19:56:54.069756 14736 rebalancer_tool.cc:1136] tablet 745e39ec23db4f90bb8b1e8b49a50244: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:54.179435 14736 rebalancer_tool.cc:1136] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: Incomplete: tablet 3e84a3865ce64eaebad4df46ec01842d, TS 58f750e4d9574c9ebfd3d9e57eefd411 -> TS c6d2c7096e404ea9a6f73b123d74c622 move failed, target replica disappeared
I20240501 19:56:54.291227 14736 rebalancer_tool.cc:1136] tablet 04a70a2ee2144f2287795ffe723d2716: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 19:56:54.307515 14736 rebalancer_tool.cc:1136] tablet 557994658e2c4c8bbc1eeacdab958ddd: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:54.359894 14736 rebalancer_tool.cc:1136] tablet cf6948589837482a9f580731e3b861b8: '2d52575d9e1b40dca0fc553d7295eb74' -> '9069ec6fcdb248479037f2d37e0e851d' move completed: OK
I20240501 19:56:54.411425 14736 rebalancer_tool.cc:1136] tablet 7e4d79f5e11b4923acb2ce543884b54a: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 19:56:54.437484 14736 rebalancer_tool.cc:1136] tablet f08535cb2f46464b9e510a45fee67b4f: 'ebbbd9ac07a94e8dbbae6def0a2df415' -> '2ca06dad25a645bca71b09741798194a' move completed: Incomplete: tablet f08535cb2f46464b9e510a45fee67b4f, TS ebbbd9ac07a94e8dbbae6def0a2df415 -> TS 2ca06dad25a645bca71b09741798194a move failed, target replica disappeared
I20240501 19:56:54.447751 14736 rebalancer_tool.cc:1136] tablet 0bf64d9f3ce94db28b032cdcd795a255: '58f750e4d9574c9ebfd3d9e57eefd411' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 19:56:54.484223 14736 rebalancer_tool.cc:1136] tablet 955edb7c12fa4b32ac58827f131931e3: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '2ca06dad25a645bca71b09741798194a' move completed: Incomplete: tablet 955edb7c12fa4b32ac58827f131931e3, TS a238fc0c6d844cedb8adccb97d01ae5e -> TS 2ca06dad25a645bca71b09741798194a move failed, target replica disappeared
I20240501 19:56:54.544474 14736 rebalancer_tool.cc:1084] tablet 61bc22cdc6ab45529066fe6f4d14d4c7: '2ca06dad25a645bca71b09741798194a' -> '1d49e80be3d04fb99b711c62d61bac09' move ignored: Not found: Peer with uuid 2ca06dad25a645bca71b09741798194a not found in consensus config
I20240501 19:56:54.544575 14736 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:54.924242 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table 109bfb23537140259f6e685179d80ba0: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
I20240501 19:56:54.983176 14736 rebalancer_tool.cc:1074] tablet 557994658e2c4c8bbc1eeacdab958ddd: '9069ec6fcdb248479037f2d37e0e851d' -> 'c6d2c7096e404ea9a6f73b123d74c622' move scheduled
I20240501 19:56:55.030423 14736 rebalancer_tool.cc:1084] tablet 3e84a3865ce64eaebad4df46ec01842d: '58f750e4d9574c9ebfd3d9e57eefd411' -> '2d52575d9e1b40dca0fc553d7295eb74' move ignored: Illegal state: Leader has not yet committed an operation in its own term
I20240501 19:56:55.030511 14736 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 19:56:55.487485 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table b4c7884870794c24aee087d8ea5702d0: could not find any suitable replica to move from server 2ca06dad25a645bca71b09741798194a to server 2d52575d9e1b40dca0fc553d7295eb74
W20240501 19:56:55.487620 14736 rebalancer_tool.cc:1205] No replica could be moved this iteration: Not found: table df4f757cd3d94896aaecbb3733403900: could not find any suitable replica to move from server 2d52575d9e1b40dca0fc553d7295eb74 to server 2ca06dad25a645bca71b09741798194a
I20240501 19:56:55.524449 14736 rebalancer_tool.cc:1074] tablet 7e4d79f5e11b4923acb2ce543884b54a: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move scheduled
I20240501 19:56:55.576488 14736 rebalancer_tool.cc:1136] tablet 7e4d79f5e11b4923acb2ce543884b54a: '58f750e4d9574c9ebfd3d9e57eefd411' -> '' move completed: OK
I20240501 19:56:55.670167 14736 rebalancer_tool.cc:1123] tablet ae2885f7933f462c96db9a99a4eb2090: '2d52575d9e1b40dca0fc553d7295eb74' -> '2ca06dad25a645bca71b09741798194a' move is abandoned: Illegal state: ae2885f7933f462c96db9a99a4eb2090: source replica 2d52575d9e1b40dca0fc553d7295eb74 does not have REPLACE attribute set
I20240501 19:56:55.685473 14736 rebalancer_tool.cc:1136] tablet 706878734216445f8422e422228d4230: 'a238fc0c6d844cedb8adccb97d01ae5e' -> '2ca06dad25a645bca71b09741798194a' move completed: OK
I20240501 19:56:55.751008 14736 rebalancer_tool.cc:1136] tablet ce6add2f19a649b3a12497951047705c: '2d52575d9e1b40dca0fc553d7295eb74' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
I20240501 19:56:57.095672 14736 rebalancer_tool.cc:1136] tablet ca3285abe3654723a6922672391e9423: '58f750e4d9574c9ebfd3d9e57eefd411' -> '1d49e80be3d04fb99b711c62d61bac09' move completed: OK
I20240501 20:01:58.351521 14736 rebalancer_tool.cc:1136] tablet 557994658e2c4c8bbc1eeacdab958ddd: '9069ec6fcdb248479037f2d37e0e851d' -> 'c6d2c7096e404ea9a6f73b123d74c622' move completed: OK
Illegal state: tablet server 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): unacceptable health status UNAVAILABLE
W20240501 20:01:58.508538 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:01:58.552670 19027 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:01:58.579205 18952 raft_consensus.cc:1050] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 to VOTER
I20240501 20:01:58.579618 18952 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 11, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } }
W20240501 20:01:58.581071 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:01:58.581130 12725 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 20:01:58.581130 12595 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 20:01:58.581655 18834 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 20:01:58.581789 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:01:58.581861 19029 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 20:01:58.583590 18834 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.584031 12725 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.585800 12595 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:01:58.586642 12448 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 14 to 15, ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 20:01:58.617758 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:01:58.683013 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 17 similar messages]
W20240501 20:01:58.683676 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:58.755955 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:58.821430 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:59.094267 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:59.148723 19027 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:01:59.216293 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 20:01:59.266233 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:59.295893 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 20:01:59.367919 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:01:59.418843 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:01:59.516968 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:01:59.688966 19064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:00.128530 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:00.164628 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240501 20:02:00.175534 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 64 similar messages]
I20240501 20:02:00.205281 19064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:00.305276 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:02:00.583261 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:00.606467 19078 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:00.679880 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:00.732885 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:00.734992 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:00.765681 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:00.838229 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:00.876370 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:02:00.907593 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 34 similar messages]
W20240501 20:02:00.908210 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:00.920775 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:00.964991 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:01.094193 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:01.153859 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:02:01.155346 19109 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:01.201138 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:01.201325 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:01.278362 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:01.508920 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:01.523950 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:02:01.554847 19109 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:01.721386 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 20:02:01.937453 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:01.979933 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:02:02.091528 19109 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:02.515455 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:02.623647 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:03.015632 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet ae2885f7933f462c96db9a99a4eb2090: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:02:03.020618 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:03.128177 19109 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:03.168017 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.247416 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.248905 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:03.276319 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.346808 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.388147 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.417794 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.472213 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.491881 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.546986 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:03.667816 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:02:03.668572 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.688630 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.714615 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:02:03.714864 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:03.715286 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:03.767045 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:02:03.956406 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:04.018826 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:04.299722 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 04ccc8addc55441b9ab9d8265b90b03b: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 20:02:04.326968 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:04.333585 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:02:04.361014 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 20:02:04.466454 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:04.506748 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:04.808012 19201 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:04.981546 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:05.277169 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:05.443982 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:05.446071 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 10 similar messages]
W20240501 20:02:05.446656 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 706878734216445f8422e422228d4230: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:02:05.453506 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:05.596161 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:05.628350 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:05.735829 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:02:05.774780 19201 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:05.785176 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:05.827759 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:05.912505 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:02:05.934813 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:05.961746 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:06.021624 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:06.101907 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:06.122092 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:06.155876 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:02:06.161412 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:06.183714 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:02:06.368691 19201 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:06.423902 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:06.440532 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:06.616737 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:02:06.832559 19201 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:06.875779 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:06.943822 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 20:02:07.248649 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:02:07.340538 19201 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:07.469535 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:07.882033 19201 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:07.914476 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:08.090538 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:08.211663 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:08.270903 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:02:08.295776 19201 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:08.331513 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:08.448222 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:08.496299 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:08.663724 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:02:08.684257 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:08.697645 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:02:08.701678 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:08.717223 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:08.761976 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:02:08.776963 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:08.780678 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:08.846030 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:08.846730 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:08.900698 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:08.922832 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:09.096370 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:02:09.233366 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:09.276620 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:09.498879 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:02:09.505965 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 20:02:09.602275 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:02:09.626730 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:09.719285 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:10.071076 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:10.198729 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:10.560381 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:10.657884 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:02:10.663530 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:10.783366 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:10.803718 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:10.975667 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:02:10.998152 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:11.007877 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:11.081986 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:02:11.094116 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:02:11.122612 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:11.126760 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:11.186304 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:11.216842 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:11.287779 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:11.334409 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:11.385808 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:11.444458 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:11.492978 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:11.648694 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:02:11.652778 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:11.756134 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240501 20:02:11.756928 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:02:11.882141 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:12.082224 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:02:12.093545 19368 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:12.177924 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 20:02:12.339031 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:12.686399 19134 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:12.887990 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:13.167835 19368 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:13.222587 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:02:13.274613 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:13.329421 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:13.383847 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:13.441416 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:13.466550 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:13.560580 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:02:13.593144 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:13.679034 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:02:13.728192 19447 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:13.799532 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:13.804185 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:02:13.814416 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:13.830188 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:13.854897 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:13.894223 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:14.015802 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:14.038065 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:02:14.098372 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:14.256351 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:02:14.315479 19447 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:14.539745 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:14.589773 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:14.630941 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 20:02:14.674623 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:02:14.784862 19447 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:15.095093 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:15.403506 19447 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:15.604571 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:15.614997 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:15.798444 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:15.828207 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:15.854403 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:02:15.927567 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:15.981779 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:02:15.987505 19447 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:15.992537 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:02:16.065891 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:16.072008 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:16.183925 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:16.194295 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:16.338824 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:02:16.487030 19513 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:16.490054 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:16.498471 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:16.558849 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:16.647439 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:16.653242 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:16.751293 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:16.974424 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:02:17.006951 19540 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:17.079046 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:17.177023 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240501 20:02:17.394572 19513 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:17.522706 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:17.844841 19513 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:17.906412 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:18.015782 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:02:18.235646 19546 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:18.235687 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:18.262650 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:02:18.292724 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:18.387346 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:18.598249 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:18.621969 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:18.626425 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:02:18.730774 19546 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:18.736696 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:18.751991 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:18.834266 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:02:18.838335 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:19.019093 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:19.104979 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:19.105675 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:02:19.141903 19546 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:19.162784 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:19.205827 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:19.270891 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:19.298960 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:19.420966 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:02:19.526007 19513 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:19.672211 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:19.711256 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:02:19.740695 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
I20240501 20:02:19.988176 19513 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:20.287600 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:20.473089 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:02:20.524091 19638 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:20.623839 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:02:20.708236 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:20.793596 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:20.903623 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:02:21.030332 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:02:21.078011 19638 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:21.092999 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:21.170775 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:02:21.171394 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:21.234799 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:21.257310 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:21.261500 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:21.273809 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:21.508221 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:02:21.555995 19513 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:21.567239 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:21.613688 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:21.729566 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:21.799201 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:21.858091 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:21.875528 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:02:22.011139 19513 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:22.191504 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 20:02:22.414888 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:22.513803 19513 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:22.866204 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:22.974340 19513 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:23.174127 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:23.276487 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:23.353216 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:23.429705 19713 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:23.523378 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:02:23.537803 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:23.579077 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:23.655674 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:23.672416 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:23.757028 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:23.775381 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:23.788894 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:23.831723 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:23.901767 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:02:23.913859 19690 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:24.041077 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:24.123983 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:24.210842 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:24.217372 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:24.340859 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:24.475983 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:24.516433 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:02:24.529745 19690 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:24.615103 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:24.630046 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 20:02:24.816191 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:02:25.045570 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:25.118839 19763 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:02:25.528900 19690 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:25.633904 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:02:25.669909 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:25.754711 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:25.795156 12437 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: ChangeConfig RPC to 127.5.139.66:36895 timed out after 30.000s (SENT) [suppressed 7 similar messages]
W20240501 20:02:25.796583 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:02:25.862222 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:02:25.908084 19763 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:26.037113 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:02:26.049099 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.078476 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:26.079003 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.116354 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.166705 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.285228 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.358196 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.425722 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:02:26.426383 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.470522 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.478396 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:26.490047 19690 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:26.705360 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.775272 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:26.876812 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:26.884889 19798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:26.931526 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:02:27.001387 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:27.110539 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:27.148074 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240501 20:02:27.392890 19826 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:27.476609 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:27.861547 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:27.958956 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:02:28.010602 19826 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:28.184134 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:28.187513 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:02:28.449012 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:28.451496 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:28.496835 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:28.503017 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:02:28.568792 19826 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:28.629346 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:28.643499 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:28.877436 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:02:28.926582 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:28.930871 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:28.940683 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:02:28.941403 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:28.989369 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:29.070655 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:29.126984 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:02:29.135318 19826 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:29.291975 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:29.478943 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:29.523561 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:29.524160 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:29.544795 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240501 20:02:29.549276 19874 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:29.637583 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:29.653144 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:02:29.855511 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:02:30.032342 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:30.039292 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:02:30.111900 19874 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:30.497603 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:30.726965 19905 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:30.727164 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:30.857295 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:31.011219 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:31.037770 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:31.089118 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:31.100369 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:31.101004 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:02:31.156759 19874 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:31.189953 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:31.260654 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:31.304749 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:31.386799 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:02:31.431646 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:02:31.432252 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:31.474959 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:02:31.680583 19874 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:31.700363 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:31.828863 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:31.889763 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:32.070289 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:32.138973 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:32.142418 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:32.159022 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240501 20:02:32.299777 19955 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:32.410012 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:32.724567 19874 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:32.922326 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:33.127501 19874 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:33.353703 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:33.413833 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:33.476981 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:33.534710 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:33.573344 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:33.576673 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:02:33.655769 19874 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:33.671340 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:33.714545 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:02:33.783051 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:33.856158 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:34.013947 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:34.014297 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:02:34.014808 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:34.029249 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:34.036372 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:02:34.138845 19874 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:34.175884 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:02:34.301213 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:34.386349 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:34.627676 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:34.645232 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:34.653118 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:02:34.758502 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:34.792150 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 20:02:34.875062 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:34.942042 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:02:35.136946 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:35.430964 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:35.709403 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:35.865420 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:35.909647 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:35.914426 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:36.030181 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:36.064335 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:02:36.190861 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:36.232918 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:36.275792 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:36.290014 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:36.314742 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:36.477658 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:36.503649 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:36.507205 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:02:36.528362 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:02:36.528980 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:36.576434 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:36.698693 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:36.757578 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:36.775461 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:36.919150 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:36.943357 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:02:37.132407 20095 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:37.280323 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:37.461511 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 20:02:37.489370 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:37.542081 20095 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:02:38.043653 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:38.074362 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:38.375752 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:38.377609 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:02:38.554167 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:38.583333 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:38.650614 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:38.733772 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:38.823220 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:38.860834 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:38.919472 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:38.953001 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:38.992826 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:39.019979 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:39.061626 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:39.064206 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:02:39.064826 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:39.124979 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:02:39.168187 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:02:39.472115 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:39.488083 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:39.553125 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:39.557760 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:39.595381 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:39.698292 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 20:02:39.795079 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:02:39.898999 20095 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:39.947650 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:02:40.069288 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:40.348165 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:40.483367 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:40.632817 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:02:40.927493 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:40.972878 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:41.038763 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:41.125126 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:41.158154 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:41.186072 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:41.344338 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:02:41.362587 20193 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:41.428349 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:41.450656 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:41.457573 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:41.467937 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:41.471917 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:41.495061 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:02:41.650696 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:02:41.651305 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:41.683207 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:02:41.912263 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:41.948673 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:42.035676 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:42.092021 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:42.249562 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:42.386976 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:02:42.484678 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:42.514619 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:42.547304 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
I20240501 20:02:42.892459 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:42.937534 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:43.253643 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:02:43.448356 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:43.519184 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:43.543058 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:43.560817 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:43.576894 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:43.828876 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:43.843187 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:02:43.910559 20260 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:43.949267 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:43.984056 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:44.009277 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:44.041911 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:44.100466 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:44.104416 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:02:44.105089 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:02:44.121977 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:02:44.316618 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:44.370934 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:44.576486 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:02:44.742611 20289 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:44.749768 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:44.813158 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:44.838017 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:45.046993 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:45.047784 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:02:45.207185 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
I20240501 20:02:45.223532 20003 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:45.234047 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:45.707160 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:45.781807 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:45.829068 20289 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:45.924744 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:45.963061 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:46.037914 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:46.077788 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:46.180516 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:46.205776 20289 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:46.349845 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:02:46.417213 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:46.473382 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:46.482141 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:46.533727 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:46.547575 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:46.587126 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:46.636453 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:46.675982 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:02:46.742292 20349 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:46.880545 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:02:47.038620 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:47.060066 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:47.191726 20289 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:47.307638 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:47.665940 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:47.669836 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:47.690846 20349 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:47.898555 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:48.052228 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240501 20:02:48.066172 20289 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:48.134253 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:48.180536 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:48.409581 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:48.533321 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:02:48.546535 20349 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:48.584306 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:48.650983 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:49.020661 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:02:49.034698 20289 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:49.040050 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:49.076400 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:49.082914 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:02:49.093847 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:49.101186 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:49.167183 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:49.191560 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:02:49.192183 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240501 20:02:49.497661 20349 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:49.609333 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:49.687407 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:49.719496 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:02:50.010653 20289 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:50.057298 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:02:50.129894 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:02:50.154263 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:50.243950 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:50.540086 20289 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:50.565450 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:50.639446 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 20:02:50.645714 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:50.685413 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:50.825960 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:50.953763 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:02:50.994302 20289 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:51.083705 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:51.188580 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:51.272974 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:51.426916 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:51.506269 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:02:51.541115 20452 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:51.576375 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:51.602499 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:02:51.620601 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:51.631552 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:02:51.693737 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:51.699055 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:51.924082 20452 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:52.168066 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:02:52.168725 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:52.276193 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:52.401336 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:02:52.410152 20452 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:52.533587 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:52.551271 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:52.795075 20452 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:52.916419 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:53.085253 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:53.118546 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 20:02:53.212245 20452 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:53.221433 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:53.391345 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:53.555737 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:53.613070 20452 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:53.799914 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:53.841532 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:53.851357 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:53.860177 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:53.863287 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:53.882066 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:02:53.990155 20548 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:54.039395 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:54.114552 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:54.125516 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:02:54.144383 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:54.224267 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:02:54.453702 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:54.591614 20452 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:54.714859 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:54.719779 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:54.829463 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:55.109624 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:02:55.192425 20574 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:55.198055 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:02:55.229584 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:55.411516 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:55.511478 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:55.633440 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:55.755455 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 20:02:55.761242 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:02:55.812652 20574 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:55.963872 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:56.214087 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:56.241214 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:56.270792 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:56.367452 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:02:56.382140 20591 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:56.402298 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:56.425129 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:56.439167 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:02:56.477602 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:56.578218 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:56.741225 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:56.863795 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:02:56.868757 20574 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:56.957912 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:56.978363 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:57.067965 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:02:57.252511 20574 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:57.376250 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:02:57.376850 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:57.778807 20591 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:57.844023 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:57.973541 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:57.982627 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:02:58.006834 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:02:58.067507 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:02:58.206041 20591 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:58.237890 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 20:02:58.566866 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:58.585880 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:02:58.649785 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:02:58.702838 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:02:58.806485 20591 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:58.819531 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:02:58.895905 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:02:58.930243 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:58.944774 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:02:58.956763 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:02:59.017190 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:02:59.112108 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:02:59.306568 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:02:59.325002 20591 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:02:59.332535 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:02:59.346729 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:02:59.675289 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:02:59.696870 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:02:59.899760 20591 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:00.079262 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:00.255474 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:03:00.275450 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:03:00.309326 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:03:00.375118 20699 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:00.385294 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:00.465126 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:00.483253 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:03:00.527815 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:00.686116 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
I20240501 20:03:00.858458 20591 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:00.969264 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:00.992684 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 20:03:00.993415 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:03:01.128890 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:03:01.221469 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:01.250067 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:03:01.316177 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:01.339978 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:03:01.374848 20739 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:01.420105 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:01.519430 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:01.593537 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:01.666215 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:01.684424 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:01.698493 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:03:01.755699 20591 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:01.956818 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:02.087283 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:03:02.130514 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:02.265560 20591 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:02.437785 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:03:02.694308 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:03:02.694905 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:02.735308 20739 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:02.811266 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:02.949576 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:03.018566 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:03.086835 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:03.104924 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
I20240501 20:03:03.190619 20739 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:03.282644 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:03.415463 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:03.422746 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:03:03.634235 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:03.700078 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:03:03.742821 20739 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:03.838196 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:03.846575 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:04.002777 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:04.029678 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:04.055436 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:04.123237 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:04.329317 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:03:04.334430 20828 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:04.354671 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:04.410502 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:03:04.689401 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:03:04.853993 20840 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:04.923566 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:05.219561 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:03:05.274467 20828 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:05.389881 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:05.418906 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 20:03:05.517616 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:05.570989 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:05.619755 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:05.653097 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 21 similar messages]
I20240501 20:03:05.805016 20828 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:05.867329 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:03:05.959862 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:05.980058 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 20:03:06.055392 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:06.079512 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:06.196270 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:03:06.339896 20828 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:06.457643 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:06.552762 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:06.558362 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:06.674484 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:06.692776 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:06.705996 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:03:06.722951 20828 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:06.723688 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:03:07.132213 20828 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:07.159515 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:07.308084 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:07.391065 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:03:07.537696 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:07.643388 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:03:07.725354 20917 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:07.772378 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:07.775034 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:03:07.895787 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:07.952910 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:08.046548 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:08.311367 20934 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:08.474099 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:08.539314 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 20:03:08.575212 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:08.593384 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:08.644618 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:08.722043 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:03:08.747622 20934 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:08.902980 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:03:09.005805 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:09.058905 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:09.140923 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:09.154086 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:09.224813 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:03:09.253538 20958 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:09.439690 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:09.451139 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:03:09.496764 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:09.709172 20958 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:09.893579 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:09.981684 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:10.240386 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:10.247779 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:03:10.283660 20934 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:10.363389 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:10.524732 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:03:10.637355 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:10.688511 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:10.815934 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:03:10.847047 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:10.852299 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:03:10.869414 20934 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:11.026997 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:03:11.046308 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 20:03:11.085291 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:11.264210 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:11.282315 20934 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:11.289654 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:11.297148 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:11.351739 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:11.384733 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:03:11.761754 20934 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:11.794061 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:11.859817 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:11.895567 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:12.232894 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:12.301488 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:12.324246 21001 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:12.413151 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:03:12.583089 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:12.721009 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:12.789664 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:03:12.790287 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:12.863435 21057 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:13.063879 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:13.082496 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:13.197098 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:13.357690 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:13.441196 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:03:13.465941 21057 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:13.624835 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:13.662207 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:13.702642 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 20:03:13.769862 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:13.771593 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:13.814431 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:13.821097 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:03:13.834697 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:03:13.864955 21073 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:14.173794 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:14.194072 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:14.261619 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:03:14.320031 21057 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:14.328132 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:14.587046 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:03:14.803551 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:14.880043 21112 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:14.944329 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:15.281347 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:15.341030 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:15.384804 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:03:15.441304 21112 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:15.484576 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:15.641778 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:03:15.665948 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:15.674286 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:15.874838 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:03:15.929771 21057 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:16.066083 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:03:16.091179 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:03:16.091909 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:16.168056 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:16.224032 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:16.239845 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 20:03:16.359105 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:03:16.377142 21057 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:16.447032 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:16.484185 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:16.626407 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:16.667865 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:16.855479 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:03:16.879935 21057 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:16.934209 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:17.002800 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:17.298871 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:17.455957 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:17.495358 21057 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:17.518944 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:03:17.634348 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:17.706795 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:17.883877 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
I20240501 20:03:17.886688 21175 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:17.904788 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:18.094774 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:18.194543 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:18.199209 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:18.310217 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:18.367707 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:03:18.451036 21175 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:18.525265 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 20:03:18.756004 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:18.807910 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:18.808346 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:18.843901 21203 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:18.998673 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:19.019023 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:19.093978 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:19.124629 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:03:19.203647 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:19.359227 21228 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:19.459641 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:19.643965 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:19.659639 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:03:19.738762 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:19.779883 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:03:19.878769 21228 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:20.340237 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:20.344321 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:20.408160 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:20.437323 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:03:20.494118 21203 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:20.703814 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240501 20:03:20.774413 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:20.842795 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:20.849145 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:20.956135 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 20:03:21.065991 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:03:21.115563 21203 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:21.273662 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:21.308913 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:21.383303 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:21.429641 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:03:21.487056 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:21.557859 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:03:21.588089 21280 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:21.611377 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:03:21.697928 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:21.854389 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:21.992322 21203 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:22.207729 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:22.270646 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:22.313596 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:22.345592 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:22.546025 21203 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:22.598886 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:03:22.761816 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:22.887447 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:22.969179 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:03:22.969743 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:23.105614 21203 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:23.141793 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:23.372148 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:23.510248 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:23.535333 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 20:03:23.589004 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:23.607453 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:03:23.637028 21203 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:23.707247 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:23.746775 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:23.852690 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:24.042912 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:03:24.107409 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:24.128178 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:03:24.142051 21203 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:24.221378 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:24.517701 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:24.631522 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:24.689464 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:03:24.753556 21375 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:24.765923 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:24.903030 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:25.019845 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:25.118830 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:25.293653 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:25.328934 21393 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:25.674844 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:25.747763 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:25.753015 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:25.762614 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:25.778671 21375 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:25.784394 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:03:25.934510 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:26.018183 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:26.190901 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:26.301774 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 20:03:26.305469 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:26.320386 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:26.378345 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:03:26.391692 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:26.465106 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:03:26.662938 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:26.676164 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:03:26.757405 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:26.806284 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:26.941337 21439 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:27.039091 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:27.247956 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:27.278879 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:27.358397 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:03:27.380661 21439 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:27.429078 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:27.675004 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:03:27.922432 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:27.977432 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:03:27.978019 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:28.004699 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:28.250612 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:28.290295 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:28.310348 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:03:28.392396 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:28.500226 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:28.595259 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:28.642045 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 20:03:28.774888 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:03:28.775619 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:28.920435 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:28.965428 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:03:29.058971 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:29.106576 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:29.123605 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:29.271147 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:03:29.394718 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:29.407260 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:29.488895 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:29.623143 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:29.719143 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:29.807888 21500 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:29.832378 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:03:29.832980 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:30.184386 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:30.298621 21500 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:30.399273 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:30.627418 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:30.731222 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:30.768720 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:30.777995 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:30.786794 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:30.792979 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:03:31.021344 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:31.186650 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:31.196607 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 20:03:31.337764 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:03:31.356509 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:31.434926 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:31.584492 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:31.610126 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:03:31.702231 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:31.710166 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:31.752136 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:03:31.769171 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:03:31.852571 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:03:31.904072 21565 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:31.972527 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:32.201521 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:32.278360 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:32.315004 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:03:32.443336 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:32.472275 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:32.887735 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:32.899909 21565 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:32.935813 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:33.056219 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:03:33.075464 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:33.121770 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:03:33.122360 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:03:33.395026 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:33.431660 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:33.454818 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:33.493986 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:33.607678 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:33.690757 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
I20240501 20:03:33.775830 21565 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:33.893847 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:34.003335 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:34.056926 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:03:34.175443 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:34.270751 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:34.323846 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:34.414187 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:34.425488 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:03:34.492319 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:34.627938 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:03:34.668233 21422 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:34.693382 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:34.892825 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:03:34.965467 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:35.000043 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:03:35.256728 21647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:35.320276 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:35.403995 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:35.416054 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:35.479130 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:35.835278 21666 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:35.921672 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:03:36.003659 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:36.063225 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:36.075934 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 20:03:36.097746 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:36.354183 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:03:36.358378 21666 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:36.434862 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:36.556860 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:36.650563 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:36.701838 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:36.786587 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:03:36.794513 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:36.826820 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:03:36.964780 21698 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:36.970110 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:37.103899 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:37.126503 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:37.271620 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:03:37.431685 21698 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:37.448150 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:37.706684 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:37.739634 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:37.798254 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:37.940865 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:03:37.988929 21698 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:38.066141 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:03:38.103351 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:38.170934 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:03:38.346197 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:03:38.466077 21725 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:38.519479 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:38.668455 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 20:03:38.685963 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:38.901921 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:38.934175 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:38.950410 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:39.013836 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:03:39.030720 21755 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:39.204211 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:39.367415 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:39.391321 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:03:39.395028 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:39.531147 21768 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:39.646927 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:39.758531 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:39.863857 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:39.918612 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:03:39.919154 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:39.974970 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:40.002555 21768 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:40.162297 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:40.187336 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:40.247613 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:40.374558 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:40.463411 21768 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:40.812525 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:40.817894 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:40.945168 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:03:40.968070 21755 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:40.991459 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:41.242080 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 20:03:41.273193 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:41.371753 21768 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:41.423982 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:41.430341 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:41.474210 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:41.527295 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:41.633229 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:41.675734 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:41.796306 21768 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:42.041172 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:42.119006 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:03:42.146909 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:42.175258 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:03:42.258733 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:42.262267 21768 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:42.407356 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:42.543383 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:42.631676 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:42.686781 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:42.686831 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:42.709318 21755 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:42.748133 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:03:43.145495 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:03:43.201320 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:03:43.276130 21755 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:43.281652 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:03:43.282274 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:43.512820 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:43.578732 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 20:03:43.620927 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:43.653964 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:03:43.685900 21755 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:43.735131 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:43.885283 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:43.898810 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:44.098014 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:03:44.190459 21755 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:44.210712 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:44.342447 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:44.526949 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:44.589258 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:03:44.652281 21892 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:44.728256 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:44.852689 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:44.861210 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:44.925019 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:03:44.928745 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:03:45.169518 21915 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:45.171540 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:45.297545 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:45.322016 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:45.352813 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:45.732868 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:45.754118 21892 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:46.051249 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 20:03:46.051925 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:46.071657 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:46.090669 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:46.191483 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240501 20:03:46.271632 21892 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:46.302878 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:46.307667 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:46.617189 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:46.628683 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:46.777550 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:03:46.851230 21892 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:46.910133 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:47.120973 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:03:47.187335 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:03:47.215245 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:47.254763 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:03:47.261361 21892 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:47.332911 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:47.333199 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:47.534611 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:47.650880 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:03:47.705777 21958 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:47.718016 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:47.719991 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:47.941740 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:48.144017 21958 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:48.240777 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:03:48.501808 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:48.512578 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:03:48.526006 21892 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:48.553669 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:03:48.554294 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:48.717494 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:48.721879 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 20:03:48.890758 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:48.930073 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:03:49.004398 21958 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:49.071676 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:49.120182 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:49.475641 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:49.556021 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:49.578318 21958 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:49.664530 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:49.666260 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:49.680685 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:03:49.750195 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:49.771716 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:49.966004 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:50.022879 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:03:50.082609 22047 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:50.090166 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:50.271363 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:50.376802 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:50.478257 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:50.662675 22047 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:51.032336 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:51.056887 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:03:51.057739 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:51.091475 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:03:51.142416 22064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:51.249248 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 20:03:51.341220 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:51.488417 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:51.494072 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:51.513465 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:51.582800 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:03:51.647495 22064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:52.006381 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:52.008390 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:52.042165 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:52.121490 22064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:52.121552 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:52.302176 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:03:52.337934 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:52.363559 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:03:52.476475 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:52.479395 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:52.675272 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:03:52.715698 22064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:52.859572 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:52.935616 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:53.042058 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:03:53.116928 22118 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:53.265131 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:03:53.404474 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:53.555850 22064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:53.571130 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:03:53.687346 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 20:03:53.729099 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:53.819707 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:53.849948 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:53.933920 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:53.967859 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:54.033270 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:03:54.112972 22064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:54.325503 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:54.354233 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:54.489068 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:54.607465 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:03:54.628706 22157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:54.656545 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:54.731338 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:54.796168 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:54.926247 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:54.952111 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:03:55.030972 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:03:55.097484 22157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:03:55.182210 12442 maintenance_manager.cc:419] P be1305e487d64b748456a721443e3b3b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033349
I20240501 20:03:55.185420 12441 maintenance_manager.cc:643] P be1305e487d64b748456a721443e3b3b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.001s Metrics: {"bytes_written":2012,"delete_count":0,"lbm_write_time_us":120,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":73,"threads_started":1,"update_count":27,"wal-append.queue_time_us":145}
I20240501 20:03:55.185770 12442 maintenance_manager.cc:419] P be1305e487d64b748456a721443e3b3b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.138720
I20240501 20:03:55.191874 12441 maintenance_manager.cc:643] P be1305e487d64b748456a721443e3b3b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.003s sys 0.003s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14651,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":290,"lbm_read_time_us":157,"lbm_reads_lt_1ms":17,"lbm_write_time_us":178,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":129,"threads_started":2,"update_count":27}
W20240501 20:03:55.295089 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:55.320842 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:55.455199 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:03:55.587838 22157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:55.821784 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:55.900100 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:56.023535 22064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:56.108486 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:03:56.154506 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 20:03:56.211445 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:56.429832 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:56.473486 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:56.497704 22157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:56.547782 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:56.617345 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:56.680462 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:56.824313 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:56.912292 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:56.922729 22064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:57.071257 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:57.120966 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:57.135922 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:57.220111 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:57.424940 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:57.451474 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:03:57.489007 22157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:57.491377 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:57.541491 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:03:57.930739 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:57.934101 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:03:57.963977 22157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:58.059882 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:58.272802 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:03:58.360390 22157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:58.372144 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:58.376183 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:03:58.611790 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:58.715713 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 20:03:58.805577 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:58.822530 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:03:58.862560 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:58.909366 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:03:58.938956 22157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:59.079378 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:59.221347 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:59.250393 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:59.330622 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:59.352368 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:03:59.376783 22288 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:03:59.473703 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:59.594043 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:03:59.651160 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:03:59.764353 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:03:59.925324 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:03:59.931301 22288 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:00.078627 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:04:00.245388 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:04:00.255553 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:00.425092 22315 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:00.524210 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:04:00.711272 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:00.738665 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:00.751427 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:00.985572 22315 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:01.007616 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:01.025367 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 20:04:01.026170 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:04:01.126960 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 20:04:01.154841 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:04:01.252560 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:01.327546 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:01.415412 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:01.514643 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:04:01.571125 22361 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:01.685657 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:01.726920 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:01.941210 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:01.989362 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:04:02.137073 22361 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:02.227344 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:02.275400 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:02.302441 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:02.353016 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:04:02.430473 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:02.448431 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:04:02.468382 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:04:02.563759 22315 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:02.669211 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:02.836246 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:03.016288 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:04:03.183715 22315 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:03.255942 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:03.269942 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:03.385298 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:03.399483 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:03.707394 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:03.708065 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 20:04:03.712738 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:03.741210 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:04:03.764353 22425 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:04.033772 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:04:04.134323 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:04.228789 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:04.280277 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:04.307303 22425 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:04.316521 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:04.404201 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:04.541100 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:04.739778 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:04.766335 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:04:04.769600 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:04:04.782471 22425 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:04.803814 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:04.808818 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:05.088171 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:04:05.198060 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:05.335820 22441 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:05.401854 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:05.594097 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:05.729763 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:05.819597 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:05.874147 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:05.887823 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:04:05.910655 22441 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:06.032984 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:06.160467 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:06.169713 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 20:04:06.237527 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:04:06.346474 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:04:06.399750 22425 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:06.624439 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:06.730273 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:06.756116 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:06.936838 22512 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:07.055560 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:07.164542 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:07.216015 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:07.284827 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:07.315090 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:07.460111 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:04:07.475518 22524 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:07.611819 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:04:07.744472 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:04:07.763193 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:07.764858 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:04:08.016111 22524 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:08.137856 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:08.180891 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:08.195930 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:08.233079 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:08.355453 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:04:08.418407 22524 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:08.528705 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:08.600896 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:08.606644 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:08.684973 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 20:04:08.860608 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:04:08.971437 22563 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:09.029883 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:09.082132 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:09.172765 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:04:09.400106 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:09.562299 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:09.586613 22563 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:09.699577 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:09.769543 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:09.820204 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:04:09.834133 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:09.957966 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:10.160164 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:10.183382 22563 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:10.203408 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:04:10.257272 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:10.261581 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:10.465617 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:10.668823 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:10.731047 22563 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:10.755635 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:10.806535 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:10.917608 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:11.052192 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:11.191996 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 20:04:11.216465 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:11.219108 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:04:11.242851 22563 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:11.318918 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 20:04:11.496956 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:04:11.809973 22643 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:11.833909 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:11.980373 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:04:12.007877 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:12.100421 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:12.198540 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:04:12.262001 22563 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:12.267472 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:12.486244 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:12.547521 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:12.601876 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:12.695459 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:04:12.771530 22670 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:12.887317 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:12.934316 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:13.070144 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:13.137539 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:13.207832 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:13.263092 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:13.359272 22563 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:13.383234 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:13.456009 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:13.644716 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:13.702220 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:13.790453 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
I20240501 20:04:13.890929 22699 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:14.068045 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:04:14.214293 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:04:14.255390 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:14.268586 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:14.461812 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:04:14.483510 22715 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:14.584714 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:04:14.675369 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:14.714728 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:14.854516 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:14.880523 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:14.985649 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:04:14.994302 22715 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:15.010299 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:15.128687 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:15.214609 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 20:04:15.244488 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:15.367059 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:04:15.404078 22715 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:15.649987 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:15.702983 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:15.798776 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:15.941603 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:04:15.996405 22759 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:16.045516 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 20:04:16.186307 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:16.232883 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:16.233523 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:16.415825 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:04:16.497376 22759 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:16.614609 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:16.937881 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:04:16.998507 22782 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:17.021220 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:17.106858 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:17.135275 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:17.214589 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:17.215322 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:17.312637 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:04:17.354918 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:17.430754 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:04:17.589208 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:17.634959 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:17.769939 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:17.889251 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:18.060434 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:04:18.214241 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:18.227421 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:18.239130 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:04:18.293399 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:18.335669 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:18.699879 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:18.721232 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:18.723230 22832 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:18.771373 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 20:04:19.157404 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:04:19.166579 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:19.178269 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:19.243045 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:04:19.243733 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:19.344058 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:19.507997 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:19.516259 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:19.594625 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:19.658516 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:19.696079 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:04:19.729326 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:04:19.763002 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:19.786288 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:19.959143 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:19.975319 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:20.053376 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:20.189235 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:20.225174 22862 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:20.257844 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:04:20.698127 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:20.698980 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:20.712653 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:20.743883 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:20.782256 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:21.181535 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:04:21.226130 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:21.382815 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:21.571705 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 20:04:21.617242 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:04:21.639321 22862 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:21.730051 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:21.919000 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:21.940333 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:21.969833 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:21.992853 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:22.059013 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:04:22.063329 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:04:22.081348 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:22.152990 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:22.209959 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:22.337641 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:22.420624 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:22.466028 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:22.501075 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:04:22.701458 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:22.981289 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:23.088897 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:23.159802 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
I20240501 20:04:23.307459 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:23.332487 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:23.339816 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:23.464890 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:23.502370 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:23.804283 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
I20240501 20:04:23.874575 22972 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:23.880333 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:24.168745 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:04:24.270891 22798 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:24.277204 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:04:24.381170 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:04:24.385679 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:24.441968 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:24.461208 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:24.491897 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:24.635705 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:24.658236 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:24.669842 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:04:24.716131 22986 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:24.716638 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:24.862537 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:24.910108 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:24.989288 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:04:25.138796 22972 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:25.162333 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:25.327808 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:25.367839 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:04:25.425343 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:25.426112 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:04:25.652768 23020 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:25.952616 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:25.983774 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:26.128463 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:04:26.186015 23037 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:26.471014 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:26.515904 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:26.556890 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240501 20:04:26.598958 23037 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:26.665913 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:04:26.813594 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:04:26.830430 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:26.874126 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:26.953001 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:26.960505 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:27.016762 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:27.116006 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:27.166803 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:04:27.178074 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:27.389091 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:27.400645 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:27.461843 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:27.496366 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:04:27.617977 23037 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:27.734938 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:27.971654 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:28.003480 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:28.075699 23037 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:28.340492 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:28.342658 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:04:28.561111 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:28.565811 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:28.784833 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:28.797377 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
I20240501 20:04:29.056331 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:29.065114 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:29.092080 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:29.277559 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:29.298811 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:29.320713 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:04:29.337019 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:29.416376 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:04:29.432387 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:29.495705 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:29.503294 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:29.584762 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:29.660935 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:04:29.775184 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:29.802474 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:29.855545 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:04:29.910872 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:30.052229 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:30.112100 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:30.227111 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:30.351758 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:04:30.393631 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:30.405388 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:04:30.634618 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:30.807174 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:30.823551 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:31.252045 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:31.361289 23167 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:31.378345 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 20:04:31.522035 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:31.626672 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:04:31.680344 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:04:31.684716 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:31.706382 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:04:31.706504 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:31.790207 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:31.798336 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:31.826076 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:31.868692 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:04:31.912330 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:32.044373 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:32.194856 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:32.197512 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:32.230774 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:32.332521 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:32.517706 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:32.572764 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:32.655525 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:32.739228 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:32.760569 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:32.877274 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:33.245561 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:33.278954 23064 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:33.306078 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:33.627770 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:33.652938 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:33.844297 23215 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:34.074257 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:34.088541 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 20:04:34.125463 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:34.152141 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:04:34.222884 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:34.244096 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:34.287132 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:34.337144 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:34.337733 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:34.391947 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:34.439334 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:04:34.463117 23215 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:34.533793 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:34.592967 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:34.804682 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:35.009558 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:04:35.015383 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:35.036931 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:04:35.071796 23267 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:35.223587 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:35.247761 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:35.269960 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:35.496796 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:04:35.572387 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:35.621101 23283 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:35.950646 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:36.102420 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:04:36.145959 23294 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:36.421623 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:36.448140 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:36.487025 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:04:36.537472 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:36.597582 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:04:36.655866 23310 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:36.691937 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 20:04:36.802873 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:04:36.852334 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:36.880960 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:36.886381 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:36.924047 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:36.937029 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:04:37.100777 23294 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:37.191725 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:37.240002 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:37.454651 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:04:37.479436 23294 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:37.540138 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:37.675514 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:37.677639 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:04:37.754745 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:37.866765 23310 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:37.962604 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:38.135607 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:38.315300 23294 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:38.480314 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:38.710832 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:38.809819 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:04:38.833166 23294 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:38.969478 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:04:38.996418 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:39.008322 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:39.214727 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:39.265995 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:39.297955 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:39.404652 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240501 20:04:39.418633 23383 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:39.496585 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:39.503291 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:04:39.533313 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:39.582827 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:39.626808 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:39.813925 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:39.817579 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:39.821594 23294 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:39.895886 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:40.060726 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:40.101444 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:40.200929 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:40.203753 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:40.232751 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:04:40.337728 23383 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:40.531117 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:04:40.752292 23294 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:40.812851 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:41.059527 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:41.217911 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:41.253185 23294 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:41.301859 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:04:41.424284 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:41.602028 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:41.625838 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 20:04:41.666541 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:41.682042 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:41.820302 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:41.827153 23451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:41.987744 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:42.003631 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:04:42.060060 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:42.118600 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:42.203536 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:42.214795 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:42.244203 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:04:42.255306 23451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:42.490891 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:42.536830 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:42.540963 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:42.679054 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:42.761562 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:04:42.822136 23451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:43.165814 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:43.264994 23294 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:43.581401 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:43.628043 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:43.748915 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:04:43.840932 23451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:44.140408 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:44.155907 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:04:44.166185 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:44.209812 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:44.251612 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 20:04:44.288033 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:44.339733 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:44.357357 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:04:44.421818 23451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:44.541261 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:44.611402 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:44.622201 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:44.666616 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:44.667384 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:44.710124 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:44.926031 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:04:44.933605 23531 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:45.004865 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:45.054641 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:45.103454 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:45.220292 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:45.320400 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:45.460271 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:45.514081 23531 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:45.720717 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:04:45.952075 23531 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:45.983078 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:46.075104 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:46.105865 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:46.436520 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:46.495759 23544 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:46.652814 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 20:04:46.724471 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:04:46.747357 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:46.769903 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:46.815225 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:46.817463 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:46.843247 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:04:46.894855 23531 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:46.956943 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:47.012210 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:47.075156 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:47.145018 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:04:47.145630 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:04:47.293161 23544 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:47.450074 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:47.465070 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:47.690528 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:47.729372 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:04:47.734712 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:04:47.761884 23544 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:47.976616 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:48.334764 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:04:48.337110 23622 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:48.405791 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:48.775538 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:04:48.830296 23544 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:48.862330 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:48.929024 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 20:04:49.021874 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:49.133374 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:49.209638 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:49.242867 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:04:49.243592 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:49.244586 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:04:49.246218 23622 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:49.288823 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:49.343045 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:49.461741 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:49.518445 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:49.536523 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:04:49.727478 23544 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:49.760180 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:49.760789 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:04:49.781708 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:50.015972 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:50.161365 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:04:50.190979 23622 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:50.282335 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:50.283749 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:50.442150 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:50.700197 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:04:50.777952 23686 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:50.818714 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:50.893720 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:04:51.025417 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:51.272023 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:51.325176 23622 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:51.410946 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:51.455600 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:51.705821 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:51.767714 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 20:04:51.824716 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:04:51.841845 23686 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:51.875434 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:04:51.888427 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:51.943939 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:52.002362 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:52.093919 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:52.097875 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:52.193385 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:52.306494 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 20:04:52.307103 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:04:52.386147 23686 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:52.569849 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:52.574100 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:52.735299 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:52.782814 23686 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:52.783373 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:53.080617 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:53.228536 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:53.253165 23728 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:53.474004 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:04:53.671161 23728 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:53.693214 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:53.806021 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:53.821810 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:54.033416 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 20:04:54.195829 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:54.288586 23728 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:54.429879 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:54.467478 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:54.488200 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:54.590715 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:54.591472 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:54.642712 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:04:54.672860 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:54.680598 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:54.707535 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:54.740046 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:04:54.772567 23775 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:54.884470 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:04:54.885171 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:04:54.975116 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:04:55.178241 23728 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:55.299402 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:55.390651 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:04:55.581101 23775 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:55.754681 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:04:55.863652 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:55.897543 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:55.974046 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:04:55.974675 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:04:56.178295 23775 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:56.281291 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:56.320305 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:56.416306 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:56.472437 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:04:56.696218 23775 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:56.808630 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 20:04:56.857630 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:56.944245 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:57.116668 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:57.133085 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:57.137355 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:04:57.237005 23775 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:57.237285 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:57.249650 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:57.278730 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:57.330461 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:04:57.344916 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:57.408892 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:57.489008 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:04:57.684645 23841 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:57.748847 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:58.131027 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:04:58.280021 23841 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:58.351332 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:58.521664 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:58.642221 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:04:58.713804 23879 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:58.725311 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:58.884136 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:58.884948 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:59.007467 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:04:59.182242 23879 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:59.344172 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:04:59.383790 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:59.418642 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 20:04:59.471427 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:04:59.541990 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:59.618134 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:04:59.726162 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:04:59.742756 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:59.785930 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:04:59.798178 23879 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:04:59.846565 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:59.863768 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:04:59.913774 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:04:59.914300 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:00.144786 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:05:00.263054 23923 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:00.305977 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:00.498692 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:05:00.773669 23879 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:00.898030 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:05:00.921599 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:01.020840 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:05:01.033878 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 20:05:01.034605 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:05:01.052413 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:01.202214 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:01.215418 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:01.267385 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:01.331646 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:01.332183 23962 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:01.585776 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:05:01.779925 23879 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:01.881141 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 20:05:01.933805 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:02.014756 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:02.037096 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:02.172394 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:05:02.210215 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:02.290640 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:02.324240 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:05:02.397068 23990 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:02.445092 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:05:02.471966 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:05:02.478209 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:02.488212 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:02.509133 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:02.622566 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:05:02.855903 23990 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:03.001644 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:03.046344 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:05:03.382660 23990 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:03.390777 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:03.402844 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:03.508381 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:03.609148 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:03.887720 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:03.890950 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:05:03.957917 23879 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:03.989471 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:04.383399 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 20:05:04.448757 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:05:04.573096 24051 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:04.576550 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:04.599092 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:04.677592 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:04.709865 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:04.777483 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:04.778113 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:05:05.014947 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:05:05.015661 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:05.022573 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:05:05.050276 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:05:05.067931 24051 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:05.148267 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:05.248991 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:05.609328 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:05:05.621695 23879 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:05.652606 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:05.934756 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:05:05.935371 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:06.034543 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 47 similar messages]
I20240501 20:05:06.107512 24051 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:06.143929 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:06.151121 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:06.218765 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:06.371755 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:05:06.636643 23879 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:06.658674 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:06.662568 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:06.811115 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 20:05:07.046622 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:07.103598 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:07.134665 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:05:07.141695 24118 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:07.273207 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:07.275180 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:07.340071 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:07.379992 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:05:07.417492 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:07.515359 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:05:07.517127 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:07.610105 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:05:07.705857 24118 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:07.894001 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:08.122352 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:08.284186 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:08.307067 24118 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:08.499384 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:08.647289 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:08.733662 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:05:08.805636 24157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:08.857975 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:08.992815 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:09.337621 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:09.363620 24157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:09.403088 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:09.517663 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 20:05:09.599207 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:09.605531 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:09.607890 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:09.642055 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:09.653859 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:05:09.746812 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:05:09.761966 24184 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:09.807441 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:09.860044 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:09.934615 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:10.053297 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:05:10.136137 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:10.187719 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:05:10.358160 24157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:10.530810 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:10.611611 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:10.870817 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:05:10.914736 24157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:10.945070 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:11.032205 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:11.068650 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:05:11.131882 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:11.138306 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
I20240501 20:05:11.292798 24222 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:11.634179 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:11.673568 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:11.873682 24157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:11.978631 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:12.012862 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 20:05:12.019470 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:12.063719 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:05:12.149449 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:12.210850 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:12.269412 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:12.434212 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:12.438560 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:12.469573 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:05:12.473908 24263 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:12.489395 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:12.742208 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:12.742261 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:12.771739 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 20:05:12.959352 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:05:13.041009 24263 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:13.161099 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:13.231992 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:13.280282 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:05:13.468276 24157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:13.540925 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:13.791777 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:13.843382 24157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:14.222034 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:14.299679 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:05:14.396567 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:14.441855 24157 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:14.445384 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 20:05:14.593087 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:14.649654 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:14.661633 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:14.761372 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:05:14.865715 24314 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:14.884160 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:14.900228 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:14.914866 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:14.936751 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:15.023944 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:15.319649 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:05:15.320259 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:15.351871 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:15.365674 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:05:15.366240 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:15.410533 24314 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:15.618274 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:15.729089 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:15.878803 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:16.002301 24314 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:16.211159 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:05:16.211747 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:16.254179 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:05:16.301476 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:16.479039 24357 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:16.759970 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:16.823093 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:16.980455 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
I20240501 20:05:16.998201 24385 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:17.025065 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:05:17.152841 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:17.154955 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:17.206780 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:17.292910 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:17.313938 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:17.407037 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:17.417347 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:17.420686 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:05:17.607924 24385 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:17.747548 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:17.798081 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:05:17.798661 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:17.865543 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:17.915231 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:18.118407 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:05:18.179098 24385 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:18.246809 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:18.274031 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:18.649209 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:18.653963 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:18.760349 24385 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:19.214588 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:05:19.218956 24419 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:19.263090 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:19.584954 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 20:05:19.596014 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:19.643918 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:05:19.656105 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:19.685182 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:19.722111 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:05:19.757752 24419 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:19.871115 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:19.884356 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:19.894150 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:19.901528 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:20.057530 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:05:20.209061 24466 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:20.282827 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:20.425757 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:05:20.426328 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:20.621237 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:05:20.651383 24419 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:20.665724 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:05:20.666244 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:20.822122 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:20.872409 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:05:21.103945 24466 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:21.110213 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:21.218681 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:21.275275 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:05:21.445605 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:21.529297 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:05:21.644650 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:21.699498 24466 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:21.986450 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 20:05:21.995668 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:22.080430 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:22.114742 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:22.185416 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:22.221746 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:05:22.247879 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:05:22.259995 24466 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:22.414772 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:22.504143 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:22.512674 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:22.529047 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:22.617396 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
I20240501 20:05:22.792860 24466 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:22.876614 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:05:22.889271 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:22.986315 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:23.092413 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:23.102653 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:23.237996 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:05:23.258989 24466 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:23.350867 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:23.495352 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:23.667145 24546 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:24.049672 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:24.105927 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:05:24.279489 24546 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:24.394419 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1011: this message will repeat every 5th retry.
W20240501 20:05:24.434227 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:24.575063 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:05:24.587805 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:24.674259 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:05:24.678550 24546 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:24.765220 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:24.851718 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:24.877792 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:24.881291 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:25.052443 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:25.095188 24574 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:25.102394 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:25.111683 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:25.451606 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:25.484773 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:05:25.485481 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:25.529247 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
I20240501 20:05:25.539093 24546 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:25.591087 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:25.636893 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:25.751685 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:05:25.906287 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
I20240501 20:05:25.924001 24546 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:26.008564 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:26.294018 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:05:26.389267 24574 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:26.518783 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:26.584339 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:26.860642 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:05:26.861217 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1016: this message will repeat every 5th retry.
I20240501 20:05:26.949440 24546 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:26.964440 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:27.102694 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 20:05:27.188632 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:27.194162 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:27.357029 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:27.399019 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:27.445839 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:27.446346 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:05:27.508275 24662 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:27.563481 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:27.649626 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:27.650153 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:27.679255 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:27.990288 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:05:27.990924 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:28.011546 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:28.012275 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:05:28.020576 24673 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:28.392302 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:28.519148 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:28.552136 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:28.566733 24691 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:28.969444 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:29.134302 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:29.135720 24691 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:29.368201 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1021: this message will repeat every 5th retry.
W20240501 20:05:29.560110 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:29.571677 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:29.594547 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:29.642853 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 20:05:29.671036 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:29.805346 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:29.825291 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:29.888715 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:29.939473 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:30.080651 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:30.122521 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:30.173735 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:30.184230 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:30.207908 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:30.347213 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:30.384377 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:30.492708 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:05:30.539449 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:30.600875 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:05:30.876761 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:05:30.877416 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:30.898005 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 20:05:30.968364 24691 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:31.041646 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:31.302505 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:31.479768 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:05:31.533179 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:05:31.592482 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:31.647315 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:05:31.808624 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1026: this message will repeat every 5th retry.
W20240501 20:05:31.861081 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:31.937685 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:05:31.988113 24691 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:32.039103 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:32.083909 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 20:05:32.233117 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:32.343997 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:32.367894 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:05:32.368688 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:32.430753 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:32.457973 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:32.574048 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:32.642153 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:32.671471 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:32.873744 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:32.883795 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:32.891115 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:05:32.980355 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:33.004199 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:05:33.004798 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:33.326215 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:33.366313 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:33.389591 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
I20240501 20:05:33.539209 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:33.848615 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:33.945178 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 20:05:33.948676 24820 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:05:34.345566 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:34.355692 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1031: this message will repeat every 5th retry.
W20240501 20:05:34.359642 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:34.466614 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 20:05:34.529870 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:34.645705 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:34.772841 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:34.790320 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:34.795293 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:34.838094 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:05:34.894672 24853 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:34.937979 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:35.138561 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:35.185539 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:05:35.371960 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:35.410099 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:35.411821 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:35.504683 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:35.527465 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:35.755064 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:05:35.879297 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:35.942837 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:05:35.965159 24853 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:36.017374 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:05:36.017990 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:36.369264 24853 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:36.447825 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:36.505201 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1036: this message will repeat every 5th retry.
W20240501 20:05:36.603166 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 20:05:36.603739 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:05:36.816534 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:36.870851 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:36.901008 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 20:05:36.991940 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:37.113399 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:05:37.220118 24853 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:37.227821 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:37.229758 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:37.271831 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:37.386008 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:37.493849 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:37.614815 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:37.625627 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
I20240501 20:05:37.696972 24853 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:37.732872 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:37.892108 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:37.971769 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:38.058009 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:38.077626 24853 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:38.096282 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:05:38.096911 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:38.223728 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:38.343050 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:05:38.506807 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:38.529747 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:39.125159 24964 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:39.130358 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:39.134507 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:39.144115 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1041: this message will repeat every 5th retry.
W20240501 20:05:39.437081 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 20:05:39.551434 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:05:39.556006 24964 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:39.627851 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:39.824441 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:39.868176 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:39.979961 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:05:39.997179 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:40.076049 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:40.148905 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:40.158785 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:40.272250 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:40.324472 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:40.347052 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:05:40.377748 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:40.539400 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:40.551620 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:40.557843 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 20:05:40.573872 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:40.674221 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:05:40.792433 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:41.152662 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:05:41.182868 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:41.184845 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:05:41.218866 24964 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:41.380262 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1046: this message will repeat every 5th retry.
W20240501 20:05:41.603721 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:41.631342 24964 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:41.682523 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:05:41.778295 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:42.101543 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 20:05:42.105886 24703 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:42.119305 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:42.186744 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:42.192025 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:42.289762 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:05:42.371472 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:42.400050 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:05:42.577913 24964 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:42.582747 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:42.582747 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:42.604187 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:42.612340 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:42.625299 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:42.644713 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:42.792577 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:43.030642 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:43.147660 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:43.151959 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240501 20:05:43.172145 24964 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:43.238453 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:43.289542 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
I20240501 20:05:43.670469 24964 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:43.683692 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:43.871954 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1051: this message will repeat every 5th retry.
W20240501 20:05:44.107268 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:44.253214 25098 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:44.391363 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 20:05:44.544536 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:44.594310 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:44.653965 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:44.777710 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 20:05:44.793951 25098 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:44.969069 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:44.985985 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:45.044827 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:45.087682 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:45.194633 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:45.201795 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:45.220043 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:45.224810 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:45.343888 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:05:45.386710 25126 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:45.440759 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:45.519321 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:45.585269 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:45.849378 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:45.870769 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:45.877964 25098 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:46.240020 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:05:46.318665 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:46.349865 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:05:46.464947 25153 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:46.572944 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1056: this message will repeat every 5th retry.
W20240501 20:05:46.868598 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:05:46.892763 25098 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:46.938630 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:46.993131 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 20:05:47.051826 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:47.140424 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:47.256036 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:47.351241 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:47.360256 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240501 20:05:47.370191 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:05:47.382747 25098 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:47.509510 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:47.528167 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:47.538275 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:47.703102 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:47.762169 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:47.799598 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:05:47.829753 25098 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:47.939966 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 20:05:47.947432 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:48.029623 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:48.195739 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:05:48.220837 25153 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:48.253190 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:05:48.439729 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:48.615033 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:48.642673 25153 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:48.717242 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1061: this message will repeat every 5th retry.
W20240501 20:05:49.015821 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:49.078218 25153 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:49.358791 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:49.588251 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:49.670866 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
I20240501 20:05:49.684787 25239 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:49.691359 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:49.759326 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:49.912469 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:49.919739 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:49.992285 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:50.002766 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:50.048384 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:50.108486 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:50.117739 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:50.263106 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
I20240501 20:05:50.283617 25255 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:50.302294 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:50.461426 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:50.590378 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:50.669420 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:05:50.670183 25239 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:50.895185 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:50.964257 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1066: this message will repeat every 5th retry.
W20240501 20:05:50.995363 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:51.260166 25239 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:51.380007 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:05:51.380587 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:51.633028 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:05:51.743937 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:05:51.863647 25301 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:51.897025 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:51.898308 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:05:52.000797 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:05:52.285782 25239 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:52.382931 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:52.438325 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:52.438484 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 20:05:52.458130 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:52.487186 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:05:52.523442 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:52.574270 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:52.623853 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:52.629391 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:52.663944 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:52.836855 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
I20240501 20:05:52.840267 25239 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:52.841738 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:53.008872 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:53.138868 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:53.204763 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:53.227830 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:53.256654 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240501 20:05:53.353073 25239 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:53.427084 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:53.448531 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1071: this message will repeat every 5th retry.
I20240501 20:05:53.898036 25358 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:53.960623 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:54.332392 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:54.373936 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:54.425139 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:54.496543 25371 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:54.819140 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 20:05:54.907927 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:54.932674 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:54.947695 25371 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:54.976893 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:55.038395 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:55.103579 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:55.179504 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:55.250764 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:55.303052 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:55.405975 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:55.406643 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
I20240501 20:05:55.420797 25371 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:55.426278 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:55.609637 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:55.690577 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:55.809329 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:55.897872 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
I20240501 20:05:55.903002 25371 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:56.007419 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1076: this message will repeat every 5th retry.
W20240501 20:05:56.201891 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:56.403959 25371 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:56.590821 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:05:56.750108 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:56.798568 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:56.802704 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:05:56.820214 25425 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:56.996106 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:05:56.996701 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:57.155737 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:57.233785 25425 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:57.461952 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:05:57.466267 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:57.510330 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:57.582304 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:57.609584 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:05:57.646095 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:05:57.671295 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 20:05:57.680042 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:57.710757 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:57.742722 25458 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:57.856105 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:05:57.997974 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:57.998802 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:05:58.031483 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:05:58.138116 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:58.142366 25425 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:58.403784 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1081: this message will repeat every 5th retry.
W20240501 20:05:58.449473 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:05:58.458904 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:05:58.546608 25425 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:58.688398 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:05:58.739861 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:59.125802 25498 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:59.310194 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:05:59.340202 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:05:59.650301 25498 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:05:59.675954 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:05:59.774273 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:05:59.807412 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:05:59.845230 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:05:59.929138 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:06:00.030277 25509 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:00.045712 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 20:06:00.108582 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:06:00.283706 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:00.289886 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:06:00.358099 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:00.448789 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:00.495012 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:00.528595 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:06:00.557544 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 20:06:00.568082 25509 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:00.902305 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:00.956403 25509 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:00.965816 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:00.998191 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:01.049513 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1086: this message will repeat every 5th retry.
W20240501 20:06:01.056201 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 20:06:01.056988 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
I20240501 20:06:01.346596 25537 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:01.500835 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:01.570134 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:01.715358 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
W20240501 20:06:01.815502 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:01.905824 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:01.943722 25537 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:01.999629 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:06:02.098870 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:02.252478 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:02.325393 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:02.340922 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:02.382777 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:06:02.429446 25537 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:02.478392 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 20:06:02.481906 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:06:02.483058 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:02.625805 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:02.628607 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:02.742074 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:02.798517 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:02.805413 25509 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:02.926229 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:03.022706 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:03.171314 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:03.200502 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:06:03.299801 25537 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:03.316082 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:03.324537 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:03.637384 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 20:06:03.647140 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:03.657474 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1091: this message will repeat every 5th retry.
I20240501 20:06:03.691401 25537 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:03.749119 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:04.177264 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:06:04.279445 25509 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:04.298604 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:04.552402 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:04.648741 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:04.702616 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:06:04.746109 25509 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:04.746475 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:04.861007 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 20:06:04.874827 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:04.884195 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:05.066087 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:05.137270 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:05.226553 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:06:05.248593 25509 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:05.368610 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:05.467631 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:05.502458 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:05.719907 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
I20240501 20:06:05.778993 25509 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:05.808349 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 20:06:05.839532 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:06.067720 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1096: this message will repeat every 5th retry.
I20240501 20:06:06.189234 25677 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:06.329408 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:06.364961 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:06.503044 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:06.746214 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:06:06.746832 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:06.775269 25677 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:06.818081 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:06:06.856899 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:07.041977 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:07.106832 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:07.233649 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 20:06:07.245334 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
I20240501 20:06:07.339057 25719 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:07.362124 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:07.377211 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:07.484091 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:07.683377 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:07.727982 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:07.780872 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:07.786590 25719 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:07.943678 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:08.023458 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:06:08.034700 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:08.211992 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:08.264714 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:08.286043 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:06:08.308756 25719 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:08.357750 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:08.411360 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1101: this message will repeat every 5th retry.
W20240501 20:06:08.706575 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:06:08.710121 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:06:08.759758 25719 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:08.804801 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:08.841732 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:09.266410 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:09.281916 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:06:09.372995 25719 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:09.518208 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:09.599929 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 20:06:09.666265 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:09.776502 25773 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:09.779661 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:09.901489 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:10.035480 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:10.047396 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:10.116225 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:10.120711 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:10.235087 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:10.317574 25773 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:10.433857 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:10.727175 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:10.745604 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:10.860538 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
I20240501 20:06:10.866689 25773 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:10.940501 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1106: this message will repeat every 5th retry.
W20240501 20:06:11.120811 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:11.147764 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:11.165625 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:06:11.337138 25827 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:11.549098 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:11.677597 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:11.728838 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:11.777508 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 29 similar messages]
I20240501 20:06:11.946254 25845 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:12.062336 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:12.135763 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:06:12.206342 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 20:06:12.226855 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:12.234802 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:06:12.321905 25827 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:12.376205 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:12.407688 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:06:12.408217 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:12.601821 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:12.673909 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:06:12.707742 25845 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:12.816593 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:12.875819 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:13.035688 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:06:13.036379 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:13.051739 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:06:13.145717 25845 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:13.156636 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:13.450595 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:13.466226 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1111: this message will repeat every 5th retry.
W20240501 20:06:13.485834 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
I20240501 20:06:13.524817 25827 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:13.672384 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:13.745131 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:06:13.835230 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:13.857340 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:06:14.072078 25827 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:14.225447 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:14.554142 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:06:14.591379 25914 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:14.630367 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:14.675662 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:14.758077 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 20:06:14.770088 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:14.851738 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:14.917343 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:15.013711 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:15.063717 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:15.206254 25914 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:15.219383 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:15.235632 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:15.350582 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:15.468250 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:15.610184 25922 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:15.669724 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:15.902977 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:15.981092 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:16.003348 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1116: this message will repeat every 5th retry.
W20240501 20:06:16.068804 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 20:06:16.188112 25922 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:16.346680 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:16.387744 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:16.512696 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:16.580552 25961 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:16.897670 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:16.961787 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:06:16.995550 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:06:17.049844 25922 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:17.071557 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:17.087967 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:17.160452 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:06:17.279989 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 20:06:17.379679 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:17.380533 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:17.421586 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:06:17.505534 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:17.654839 26002 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:17.683183 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:17.797818 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:17.961634 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:17.964363 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 20:06:18.129732 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:18.130410 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:18.137004 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:06:18.230670 26002 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:18.307623 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:18.391079 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:18.403923 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:18.465721 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1121: this message will repeat every 5th retry.
W20240501 20:06:18.687528 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:06:18.713763 26002 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:18.780450 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:06:18.922165 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:18.948733 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:06:19.195785 26013 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:19.232091 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:19.451443 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:19.523363 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:06:19.573719 26002 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:19.581413 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:19.860805 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:19.976671 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:20.034612 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:20.098978 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:20.101496 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
I20240501 20:06:20.159790 26002 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:20.314802 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:20.321177 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:20.414074 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:20.611385 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:20.643332 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:20.653879 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:06:20.690791 26083 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:20.778327 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:20.807193 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:21.110509 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1126: this message will repeat every 5th retry.
W20240501 20:06:21.268064 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
I20240501 20:06:21.302173 26083 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:21.337215 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:21.621587 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:06:21.726840 26099 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:21.826050 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:21.902216 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:21.928166 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:22.066103 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:22.255789 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:06:22.286825 26126 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:22.298755 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 32 similar messages]
W20240501 20:06:22.338966 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:22.460033 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:06:22.478751 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 20:06:22.628391 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:22.647101 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:22.703092 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
I20240501 20:06:22.724609 26099 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:22.869923 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:22.959924 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:23.009238 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:23.090576 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:23.110585 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:23.173832 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:23.234406 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:23.235049 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:23.257117 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:23.281725 26153 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:23.465312 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:23.593976 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1131: this message will repeat every 5th retry.
I20240501 20:06:23.868670 26172 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:23.878464 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:23.895421 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:06:24.119915 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:24.233920 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:24.261422 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:24.359504 26172 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:24.434827 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:24.585626 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:24.728428 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:24.963956 26172 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:25.022922 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:25.153187 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 20:06:25.163242 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:25.167999 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:25.284047 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:06:25.421025 26172 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:25.453394 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:25.571954 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:25.614230 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:25.639457 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:25.670310 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:25.723752 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:25.788902 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:25.858752 26172 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:25.884514 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1136: this message will repeat every 5th retry.
W20240501 20:06:26.053117 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:26.349781 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:26.449787 26172 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:26.644294 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:26.816990 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:26.971832 26254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:27.017835 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:27.108443 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:27.219154 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:27.287226 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:27.404718 26254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:27.484812 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:27.512692 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:06:27.563498 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:27.581897 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:27.605721 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:27.641991 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
I20240501 20:06:27.857538 26172 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:27.882730 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:27.883901 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:27.905126 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 20:06:27.921674 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:28.096199 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:28.255581 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:06:28.369671 26254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:28.373770 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:28.374497 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1141: this message will repeat every 5th retry.
W20240501 20:06:28.390965 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:28.395886 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:28.439371 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:28.503383 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
I20240501 20:06:28.963445 26304 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:28.968773 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:28.986346 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:06:29.115273 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:29.352420 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:29.407266 26254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:29.670377 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:29.702567 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:29.710435 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:29.760134 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:29.798121 26254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:30.054522 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:30.072831 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:30.084602 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:30.227485 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 20:06:30.284732 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:30.305157 26345 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:30.367765 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:30.502460 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:30.626175 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:30.717862 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:30.744284 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:30.786029 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:06:30.817291 26357 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:30.856668 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1146: this message will repeat every 5th retry.
W20240501 20:06:30.886023 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:31.100240 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:31.306007 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
I20240501 20:06:31.342797 26375 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:31.591674 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:31.678848 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:31.694473 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:06:31.948794 26392 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:32.206272 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:32.372867 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:32.407615 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:32.423326 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:32.471556 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:32.520332 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:06:32.521029 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:06:32.548164 26409 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:32.548528 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:32.574378 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 20:06:32.631498 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:32.762980 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:06:32.763582 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:32.918008 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:06:32.944408 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:32.946617 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:06:32.995074 26409 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:33.086971 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:33.136106 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1151: this message will repeat every 5th retry.
W20240501 20:06:33.255211 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:33.292743 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:33.472501 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:06:33.605229 26442 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:33.725980 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:33.734895 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:06:33.782321 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:34.009603 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:06:34.027163 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 20:06:34.172837 26454 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:34.296958 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:34.655409 26454 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:34.704631 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:34.839241 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:34.894187 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:34.965395 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:35.036494 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:35.039337 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:35.075035 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:35.080168 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:35.199748 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
I20240501 20:06:35.224561 26487 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:35.374051 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 20:06:35.468858 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:35.473533 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:35.483366 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:35.575033 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1156: this message will repeat every 5th retry.
W20240501 20:06:35.621014 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:35.748979 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 20:06:35.848739 26503 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:35.948047 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:36.082777 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:36.368808 26503 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:36.405870 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:36.496193 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:36.680544 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:36.921222 26532 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:37.113998 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:37.199699 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:37.237357 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:37.342600 26532 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:37.393014 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:37.540776 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 20:06:37.574010 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:37.674227 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:37.682122 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:37.756686 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:06:37.854835 26532 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:37.879249 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:37.889318 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 20:06:37.966706 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1161: this message will repeat every 5th retry.
W20240501 20:06:37.976004 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:38.010892 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:38.011569 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:38.031504 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:06:38.032217 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:38.230180 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:38.277882 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:38.380203 26569 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:38.403380 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:38.679483 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:38.899700 26580 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:38.927255 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240501 20:06:38.962118 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:39.070559 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 20:06:39.092043 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:39.195302 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:39.324515 26580 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:39.451066 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:39.658499 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:39.695096 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:39.738983 26569 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:39.995123 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:40.108001 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:40.120952 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:06:40.126339 26580 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:40.149300 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:40.273885 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1166: this message will repeat every 5th retry.
W20240501 20:06:40.357331 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 20:06:40.403155 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:40.424013 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:40.483842 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:40.492360 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:40.599521 26580 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:40.651820 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:40.753561 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:40.821066 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:40.924296 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:06:40.977859 26569 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:40.996892 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:41.381007 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:41.404762 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:06:41.445561 26580 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:41.557967 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 20:06:41.839751 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:06:41.893285 26580 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:41.970155 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:41.990892 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
I20240501 20:06:42.327714 26569 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:42.449820 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:42.553997 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:42.612062 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 20:06:42.612798 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:42.656271 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:42.660562 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1171: this message will repeat every 5th retry.
W20240501 20:06:42.858318 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:42.885833 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:42.908870 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
I20240501 20:06:42.916301 26686 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:42.957491 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:43.103783 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:43.104437 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:43.122804 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:43.283085 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:43.305423 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:06:43.305987 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:06:43.327246 26686 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:43.531486 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:43.635025 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
I20240501 20:06:43.898010 26686 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:43.934804 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:44.088254 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:44.104311 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 20:06:44.192158 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:44.214155 14615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240501 20:06:44.322710 26713 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:44.508849 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:44.581419 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:44.687350 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:44.897938 26735 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:45.114564 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:45.137454 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:45.171576 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:45.188650 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 20:06:45.208972 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:45.241092 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1176: this message will repeat every 5th retry.
W20240501 20:06:45.249233 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:06:45.407867 26735 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:45.601780 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:45.645409 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:45.710533 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:45.734069 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:45.935089 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:06:45.957901 26767 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:45.976133 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:46.196244 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:46.372828 26713 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:46.380791 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:46.424705 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:46.644059 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:46.809140 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:46.840003 26767 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:47.081902 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:47.188979 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:47.225078 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:06:47.407960 26803 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:47.551597 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 20:06:47.571125 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:47.593325 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1181: this message will repeat every 5th retry.
W20240501 20:06:47.610391 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:47.726033 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:06:47.786063 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:47.819689 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:06:47.876379 26767 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:47.934481 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:48.011111 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:48.044042 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:48.135529 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:48.158419 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:48.263417 26803 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:48.326438 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:06:48.401396 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 28 similar messages]
W20240501 20:06:48.439553 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:48.537091 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:48.654821 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:06:48.691322 26767 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:48.837783 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 20:06:49.028245 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:49.094038 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:49.136335 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 48 similar messages]
I20240501 20:06:49.234668 26848 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:49.395188 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:49.435683 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 20:06:49.508834 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:49.570142 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:49.661206 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
I20240501 20:06:49.678751 26767 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:49.714656 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:49.880995 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:50.058259 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:50.140746 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1186: this message will repeat every 5th retry.
I20240501 20:06:50.286098 26875 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:50.326639 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:06:50.332645 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:06:50.403470 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:50.567319 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:50.618618 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:50.653489 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:50.722404 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:06:50.895045 26875 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:50.908231 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:06:50.960431 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:06:51.174248 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:51.339444 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 20:06:51.391988 26875 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:51.702198 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:06:51.752085 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:51.854545 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:06:51.937430 26891 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:51.960300 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:06:52.168978 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 20:06:52.266156 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:06:52.279181 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:52.348917 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:06:52.442406 26891 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:52.678145 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:52.733880 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 20:06:52.741994 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1191: this message will repeat every 5th retry.
W20240501 20:06:52.855927 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:06:52.857331 26891 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:52.888888 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:52.973338 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:53.235489 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:06:53.281098 26891 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:53.282501 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:53.306738 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:06:53.330731 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:53.377775 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:53.492561 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:53.525898 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 30 similar messages]
W20240501 20:06:53.526492 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:53.726502 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 20:06:53.844616 26891 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:53.857620 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:54.147930 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:54.187541 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240501 20:06:54.388454 26891 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:54.462076 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:54.490533 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:54.545011 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240501 20:06:54.605527 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:54.720980 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 20:06:54.790382 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:54.853967 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:06:54.940243 26998 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:54.946707 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:55.043489 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:55.081385 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:06:55.273185 14615 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1196: this message will repeat every 5th retry.
I20240501 20:06:55.326824 26586 consensus_queue.cc:579] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.015s)
W20240501 20:06:55.328146 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:06:55.332886 12855 consensus_queue.cc:237] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 20:06:55.334851 12988 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 20:06:55.335016 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.335249 27013 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 20:06:55.338302 27009 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
W20240501 20:06:55.338584 12790 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): Couldn't send request to peer 58f750e4d9574c9ebfd3d9e57eefd411. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04a70a2ee2144f2287795ffe723d2716. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.339948 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 04a70a2ee2144f2287795ffe723d2716 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:55.340117 12988 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 20:06:55.340865 12451 catalog_manager.cc:5484] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 8 to 9, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:06:55.342792 26998 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:06:55.376010 26746 consensus_queue.cc:579] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.245s)
W20240501 20:06:55.377178 12531 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:55.381264 12592 consensus_queue.cc:237] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } }
I20240501 20:06:55.382681 12988 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:06:55.382776 14680 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:06:55.383133 26746 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:06:55.383212 27021 consensus_queue.cc:579] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.154s)
I20240501 20:06:55.383392 26746 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:06:55.385340 27021 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:06:55.385906 12988 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:06:55.385973 14680 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:06:55.386523 12592 consensus_queue.cc:237] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 20:06:55.387543 12439 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 22b272a8582d4060a6a527ac4fce9ae5 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:06:55.388721 12449 catalog_manager.cc:5484] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 9 to 10, NON_VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } }
I20240501 20:06:55.388913 14680 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 20:06:55.389137 12528 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467): Couldn't send request to peer a238fc0c6d844cedb8adccb97d01ae5e. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.389298 27025 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:06:55.390062 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.391253 26746 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
W20240501 20:06:55.391810 12440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 8 similar messages]
W20240501 20:06:55.392830 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:55.392836 14680 raft_consensus.cc:2932] T d940b16d07d54945abe4a10a11dd6a2a P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } } }
I20240501 20:06:55.393373 12439 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d940b16d07d54945abe4a10a11dd6a2a with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:55.393771 12592 consensus_queue.cc:237] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 20:06:55.393787 12451 catalog_manager.cc:5484] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 9 to 10, NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:06:55.395435 14680 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:06:55.395552 12988 raft_consensus.cc:1261] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:06:55.395835 27025 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:06:55.396735 12528 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22b272a8582d4060a6a527ac4fce9ae5. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.396818 27021 consensus_queue.cc:1035] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:06:55.398025 27025 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 20:06:55.398432 14680 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 20:06:55.398861 12988 raft_consensus.cc:2932] T 22b272a8582d4060a6a527ac4fce9ae5 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 20:06:55.399565 12439 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 22b272a8582d4060a6a527ac4fce9ae5 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:55.400801 12445 catalog_manager.cc:5484] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 10 to 11, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:06:55.551616 27037 consensus_queue.cc:579] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.002s)
I20240501 20:06:55.557282 12988 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 20:06:55.558919 12855 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240501 20:06:55.559062 12921 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895): Couldn't send request to peer 58f750e4d9574c9ebfd3d9e57eefd411. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:55.559309 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.559427 27038 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:06:55.561268 27037 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 20:06:55.561730 12855 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } } }
I20240501 20:06:55.562613 12440 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ca3285abe3654723a6922672391e9423 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:55.563829 12449 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 9 to 10, NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) added. New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:06:55.564324 26891 consensus_queue.cc:579] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.256s)
W20240501 20:06:55.565583 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:06:55.567354 12725 consensus_queue.cc:237] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:06:55.568897 14286 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 20:06:55.568979 12661 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745): Couldn't send request to peer ebbbd9ac07a94e8dbbae6def0a2df415. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 800d5f43cd1b4fb6bd8bbcac262aeb80. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.569293 27043 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:06:55.570194 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.572324 26998 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } } }
I20240501 20:06:55.572762 14286 raft_consensus.cc:2932] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 9069ec6fcdb248479037f2d37e0e851d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } } }
I20240501 20:06:55.573688 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:55.574687 12451 catalog_manager.cc:5484] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 9 to 10, NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:06:55.619114 26925 consensus_queue.cc:579] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.040s)
I20240501 20:06:55.685636 26891 consensus_queue.cc:579] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.359s)
W20240501 20:06:55.687080 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:06:55.688302 12725 consensus_queue.cc:237] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
I20240501 20:06:55.689827 12592 raft_consensus.cc:1261] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 20:06:55.690280 26998 consensus_queue.cc:1035] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 20:06:55.690457 12661 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 32a17c2e392a49b7b181fa7a25a778a1. This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:55.690637 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.692008 26891 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 20:06:55.692399 12592 raft_consensus.cc:2932] T 32a17c2e392a49b7b181fa7a25a778a1 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 20:06:55.693358 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 32a17c2e392a49b7b181fa7a25a778a1 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:55.694391 12445 catalog_manager.cc:5484] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 6 to 7, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:06:55.760486 26998 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:06:55.782104 27053 ts_tablet_manager.cc:902] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
I20240501 20:06:55.782770 27053 tablet_copy_client.cc:308] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.65:38745
I20240501 20:06:55.783351 12614 tablet_copy_service.cc:140] P ebbbd9ac07a94e8dbbae6def0a2df415: Received BeginTabletCopySession request for tablet d940b16d07d54945abe4a10a11dd6a2a from peer a238fc0c6d844cedb8adccb97d01ae5e ({username='slave'} at 127.5.139.67:60523)
I20240501 20:06:55.783483 12614 tablet_copy_service.cc:161] P ebbbd9ac07a94e8dbbae6def0a2df415: Beginning new tablet copy session on tablet d940b16d07d54945abe4a10a11dd6a2a from peer a238fc0c6d844cedb8adccb97d01ae5e at {username='slave'} at 127.5.139.67:60523: session id = a238fc0c6d844cedb8adccb97d01ae5e-d940b16d07d54945abe4a10a11dd6a2a
I20240501 20:06:55.784478 12614 tablet_copy_source_session.cc:205] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:06:55.785238 27053 ts_tablet_manager.cc:1875] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:06:55.787137 27053 ts_tablet_manager.cc:1888] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 20:06:55.787320 27053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d940b16d07d54945abe4a10a11dd6a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:06:55.788393 27053 tablet_copy_client.cc:783] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Starting download of 5 data blocks...
I20240501 20:06:55.793138 27053 tablet_copy_client.cc:647] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Starting download of 1 WAL segments...
I20240501 20:06:55.794274 27053 tablet_copy_client.cc:523] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:06:55.795449 27053 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap starting.
I20240501 20:06:55.802982 27053 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:06:55.803222 27053 tablet_bootstrap.cc:492] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Bootstrap complete.
I20240501 20:06:55.803377 27053 ts_tablet_manager.cc:1367] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240501 20:06:55.803778 27053 raft_consensus.cc:348] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 20:06:55.803880 27053 raft_consensus.cc:729] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Initialized, Role: LEARNER
I20240501 20:06:55.803980 27053 consensus_queue.cc:260] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: NON_VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: true } }
I20240501 20:06:55.804224 27053 ts_tablet_manager.cc:1398] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:06:55.804692 12614 tablet_copy_service.cc:342] P ebbbd9ac07a94e8dbbae6def0a2df415: Request end of tablet copy session a238fc0c6d844cedb8adccb97d01ae5e-d940b16d07d54945abe4a10a11dd6a2a received from {username='slave'} at 127.5.139.67:60523
I20240501 20:06:55.804780 12614 tablet_copy_service.cc:433] P ebbbd9ac07a94e8dbbae6def0a2df415: ending tablet copy session a238fc0c6d844cedb8adccb97d01ae5e-d940b16d07d54945abe4a10a11dd6a2a on tablet d940b16d07d54945abe4a10a11dd6a2a with peer a238fc0c6d844cedb8adccb97d01ae5e
I20240501 20:06:55.814021 27013 consensus_queue.cc:579] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.429s)
W20240501 20:06:55.815212 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 20:06:55.816273 12855 consensus_queue.cc:237] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 20:06:55.817682 14680 raft_consensus.cc:1261] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 20:06:55.817765 12790 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bf64d9f3ce94db28b032cdcd795a255. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.818079 26586 consensus_queue.cc:1035] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:06:55.818240 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.819856 27009 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 20:06:55.820267 14679 raft_consensus.cc:2932] T 0bf64d9f3ce94db28b032cdcd795a255 P c6d2c7096e404ea9a6f73b123d74c622 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 20:06:55.821208 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0bf64d9f3ce94db28b032cdcd795a255 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:55.822572 12451 catalog_manager.cc:5484] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 9 to 10, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:06:55.855885 12904 maintenance_manager.cc:419] P a238fc0c6d844cedb8adccb97d01ae5e: Scheduling UndoDeltaBlockGCOp(d940b16d07d54945abe4a10a11dd6a2a): 4582 bytes on disk
I20240501 20:06:55.856408 12794 maintenance_manager.cc:643] P a238fc0c6d844cedb8adccb97d01ae5e: UndoDeltaBlockGCOp(d940b16d07d54945abe4a10a11dd6a2a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240501 20:06:55.881283 27025 consensus_queue.cc:579] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.518s)
I20240501 20:06:55.883882 12592 consensus_queue.cc:237] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
W20240501 20:06:55.885401 12528 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.885432 12988 raft_consensus.cc:1261] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 20:06:55.886127 27025 consensus_queue.cc:1035] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 20:06:55.886682 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:55.887923 27031 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 20:06:55.888406 12988 raft_consensus.cc:2932] T 7e4d79f5e11b4923acb2ce543884b54a P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 20:06:55.889740 12439 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 7e4d79f5e11b4923acb2ce543884b54a with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:55.891116 12445 catalog_manager.cc:5484] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 14 to 15, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 2 leader_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 20:06:55.900077 26998 consensus_queue.cc:579] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.102s)
I20240501 20:06:55.914363 26998 consensus_queue.cc:579] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.117s)
I20240501 20:06:55.941650 27068 ts_tablet_manager.cc:902] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: Initiating tablet copy from peer a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467)
I20240501 20:06:55.942153 27068 tablet_copy_client.cc:308] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.67:46467
I20240501 20:06:55.942580 12877 tablet_copy_service.cc:140] P a238fc0c6d844cedb8adccb97d01ae5e: Received BeginTabletCopySession request for tablet 04a70a2ee2144f2287795ffe723d2716 from peer 58f750e4d9574c9ebfd3d9e57eefd411 ({username='slave'} at 127.5.139.66:33687)
I20240501 20:06:55.942692 12877 tablet_copy_service.cc:161] P a238fc0c6d844cedb8adccb97d01ae5e: Beginning new tablet copy session on tablet 04a70a2ee2144f2287795ffe723d2716 from peer 58f750e4d9574c9ebfd3d9e57eefd411 at {username='slave'} at 127.5.139.66:33687: session id = 58f750e4d9574c9ebfd3d9e57eefd411-04a70a2ee2144f2287795ffe723d2716
I20240501 20:06:55.943603 12877 tablet_copy_source_session.cc:205] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:06:55.944469 27068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a70a2ee2144f2287795ffe723d2716. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:06:55.946269 27068 tablet_copy_client.cc:783] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 5 data blocks...
I20240501 20:06:55.950531 27068 tablet_copy_client.cc:647] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 1 WAL segments...
I20240501 20:06:55.951685 27068 tablet_copy_client.cc:523] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:06:55.952670 27068 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 20:06:55.959666 27068 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:06:55.959892 27068 tablet_bootstrap.cc:492] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap complete.
I20240501 20:06:55.960037 27068 ts_tablet_manager.cc:1367] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 20:06:55.960386 27068 raft_consensus.cc:348] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 20:06:55.960479 27068 raft_consensus.cc:729] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: LEARNER
I20240501 20:06:55.960589 27068 consensus_queue.cc:260] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 20:06:55.960840 27068 ts_tablet_manager.cc:1398] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 20:06:55.961315 12877 tablet_copy_service.cc:342] P a238fc0c6d844cedb8adccb97d01ae5e: Request end of tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-04a70a2ee2144f2287795ffe723d2716 received from {username='slave'} at 127.5.139.66:33687
I20240501 20:06:55.961397 12877 tablet_copy_service.cc:433] P a238fc0c6d844cedb8adccb97d01ae5e: ending tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-04a70a2ee2144f2287795ffe723d2716 on tablet 04a70a2ee2144f2287795ffe723d2716 with peer 58f750e4d9574c9ebfd3d9e57eefd411
I20240501 20:06:55.967283 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(04a70a2ee2144f2287795ffe723d2716): 4834 bytes on disk
I20240501 20:06:55.967650 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(04a70a2ee2144f2287795ffe723d2716) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
I20240501 20:06:55.970101 27074 ts_tablet_manager.cc:902] T 22b272a8582d4060a6a527ac4fce9ae5 P 9069ec6fcdb248479037f2d37e0e851d: Initiating tablet copy from peer ebbbd9ac07a94e8dbbae6def0a2df415 (127.5.139.65:38745)
F20240501 20:06:55.970271 27074 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.queue_time_us, tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.queue_time_us) (tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_cpu_time_us, tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_cpu_time_us) (tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_wall_time_us, tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_wall_time_us) (tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.queue_time_us, tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.queue_time_us) (tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_cpu_time_us, tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_cpu_time_us) (tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_wall_time_us, tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_wall_time_us) (tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.queue_time_us, tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.queue_time_us) (tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.queue_time_us, tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.queue_time_us) (tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_cpu_time_us, tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_cpu_time_us) (tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_wall_time_us, tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_wall_time_us) (tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.queue_time_us, tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.queue_time_us) (tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_cpu_time_us, tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_cpu_time_us) (tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_wall_time_us, tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_wall_time_us) (tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.queue_time_us, tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.queue_time_us) (tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.run_cpu_time_us, tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.run_cpu_time_us) (tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.run_wall_time_us, tablet-download-pool-61bc22cdc6ab45529066fe6f4d14d4c7.run_wall_time_us) (tablet-download-pool-706878734216445f8422e422228d4230.queue_time_us, tablet-download-pool-706878734216445f8422e422228d4230.queue_time_us) (tablet-download-pool-706878734216445f8422e422228d4230.run_cpu_time_us, tablet-download-pool-706878734216445f8422e422228d4230.run_cpu_time_us) (tablet-download-pool-706878734216445f8422e422228d4230.run_wall_time_us, tablet-download-pool-706878734216445f8422e422228d4230.run_wall_time_us) (tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.queue_time_us, tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.queue_time_us) (tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.run_cpu_time_us, tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.run_cpu_time_us) (tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.run_wall_time_us, tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.run_wall_time_us) (tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.queue_time_us, tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.queue_time_us) (tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.run_cpu_time_us, tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.run_cpu_time_us) (tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.run_wall_time_us, tablet-download-pool-7d7ea564ea504163a1e17561be9a5130.run_wall_time_us) (tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.queue_time_us, tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.queue_time_us) (tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.run_cpu_time_us, tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.run_cpu_time_us) (tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.run_wall_time_us, tablet-download-pool-7e4d79f5e11b4923acb2ce543884b54a.run_wall_time_us) (tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.queue_time_us, tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.queue_time_us) (tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.run_cpu_time_us, tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.run_cpu_time_us) (tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.run_wall_time_us, tablet-download-pool-800d5f43cd1b4fb6bd8bbcac262aeb80.run_wall_time_us) (tablet-download-pool-c7ddb6d79d384d39a709b5824f936d53.queue_time_us, tablet-download-pool-c7ddb6d79d384d39a709b5824f936d53.queue_time_us) (tablet-download-pool-c7ddb6d79d384d39a709b5824f936d53.run_cpu_time_us, tablet-download-pool-c7ddb6d79d384d39a709b5824f936d53.run_cpu_time_us) (tablet-download-pool-c7ddb6d79d384d39a709b5824f936d53.run_wall_time_us, tablet-download-pool-c7ddb6d79d384d39a709b5824f936d53.run_wall_time_us) (tablet-download-pool-cf6948589837482a9f580731e3b861b8.queue_time_us, tablet-download-pool-cf6948589837482a9f580731e3b861b8.queue_time_us) (tablet-download-pool-cf6948589837482a9f580731e3b861b8.run_cpu_time_us, tablet-download-pool-cf6948589837482a9f580731e3b861b8.run_cpu_time_us) (tablet-download-pool-cf6948589837482a9f580731e3b861b8.run_wall_time_us, tablet-download-pool-cf6948589837482a9f580731e3b861b8.run_wall_time_us) (tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.queue_time_us, tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.queue_time_us) (tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.run_cpu_time_us, tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.run_cpu_time_us) (tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.run_wall_time_us, tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.run_wall_time_us) (tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.queue_time_us, tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.queue_time_us) (tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.run_cpu_time_us, tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.run_cpu_time_us) (tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.run_wall_time_us, tablet-download-pool-f08535cb2f46464b9e510a45fee67b4f.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7ffbd2cbbdcd google::LogMessage::Fail() at ??:0
@ 0x7ffbd2cbfb93 google::LogMessage::SendToLog() at ??:0
@ 0x7ffbd2cbb7cc google::LogMessage::Flush() at ??:0
@ 0x7ffbd2cbcf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ffbd4125044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7ffbd4109f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7ffbd4107efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7ffbd6cba77a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7ffbd6cc3a45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7ffbd6d4476f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7ffbd6d584f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7ffbd6d43226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7ffbd6d52c0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56107549775c std::function<>::operator()() at ??:0
@ 0x7ffbd410ce06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ffbd410d6ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ffbd410f03b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x56107549775c std::function<>::operator()() at ??:0
@ 0x7ffbd40fc807 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ffbd493a6db start_thread at ??:0
@ 0x7ffbd1e7271f clone at ??:0
I20240501 20:06:56.015583 27086 ts_tablet_manager.cc:902] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 20:06:56.016193 27086 tablet_copy_client.cc:308] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.66:36895
I20240501 20:06:56.016763 12746 tablet_copy_service.cc:140] P 58f750e4d9574c9ebfd3d9e57eefd411: Received BeginTabletCopySession request for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 from peer ebbbd9ac07a94e8dbbae6def0a2df415 ({username='slave'} at 127.5.139.65:41223)
I20240501 20:06:56.016853 12746 tablet_copy_service.cc:161] P 58f750e4d9574c9ebfd3d9e57eefd411: Beginning new tablet copy session on tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 from peer ebbbd9ac07a94e8dbbae6def0a2df415 at {username='slave'} at 127.5.139.65:41223: session id = ebbbd9ac07a94e8dbbae6def0a2df415-800d5f43cd1b4fb6bd8bbcac262aeb80
I20240501 20:06:56.017814 12746 tablet_copy_source_session.cc:205] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:06:56.018883 27086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 800d5f43cd1b4fb6bd8bbcac262aeb80. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:06:56.022610 27086 tablet_copy_client.cc:783] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Starting download of 5 data blocks...
I20240501 20:06:56.028187 27086 tablet_copy_client.cc:647] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Starting download of 1 WAL segments...
I20240501 20:06:56.029606 27086 tablet_copy_client.cc:523] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:06:56.031458 27086 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap starting.
I20240501 20:06:56.045559 27086 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:06:56.045881 27086 tablet_bootstrap.cc:492] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: Bootstrap complete.
I20240501 20:06:56.046084 27086 ts_tablet_manager.cc:1367] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent bootstrapping tablet: real 0.015s user 0.015s sys 0.000s
I20240501 20:06:56.046732 27086 raft_consensus.cc:348] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:06:56.046865 27086 raft_consensus.cc:729] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ebbbd9ac07a94e8dbbae6def0a2df415, State: Initialized, Role: LEARNER
I20240501 20:06:56.046967 27086 consensus_queue.cc:260] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: NON_VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: true } }
I20240501 20:06:56.047461 27086 ts_tablet_manager.cc:1398] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 20:06:56.048058 12746 tablet_copy_service.cc:342] P 58f750e4d9574c9ebfd3d9e57eefd411: Request end of tablet copy session ebbbd9ac07a94e8dbbae6def0a2df415-800d5f43cd1b4fb6bd8bbcac262aeb80 received from {username='slave'} at 127.5.139.65:41223
I20240501 20:06:56.048157 12746 tablet_copy_service.cc:433] P 58f750e4d9574c9ebfd3d9e57eefd411: ending tablet copy session ebbbd9ac07a94e8dbbae6def0a2df415-800d5f43cd1b4fb6bd8bbcac262aeb80 on tablet 800d5f43cd1b4fb6bd8bbcac262aeb80 with peer ebbbd9ac07a94e8dbbae6def0a2df415
I20240501 20:06:56.052511 27068 ts_tablet_manager.cc:902] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Initiating tablet copy from peer 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 20:06:56.052948 27068 tablet_copy_client.cc:308] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Beginning tablet copy session from remote peer at address 127.5.139.68:41997
I20240501 20:06:56.053454 13007 tablet_copy_service.cc:140] P 2d52575d9e1b40dca0fc553d7295eb74: Received BeginTabletCopySession request for tablet ca3285abe3654723a6922672391e9423 from peer 58f750e4d9574c9ebfd3d9e57eefd411 ({username='slave'} at 127.5.139.66:43757)
I20240501 20:06:56.053568 13007 tablet_copy_service.cc:161] P 2d52575d9e1b40dca0fc553d7295eb74: Beginning new tablet copy session on tablet ca3285abe3654723a6922672391e9423 from peer 58f750e4d9574c9ebfd3d9e57eefd411 at {username='slave'} at 127.5.139.66:43757: session id = 58f750e4d9574c9ebfd3d9e57eefd411-ca3285abe3654723a6922672391e9423
I20240501 20:06:56.054988 13007 tablet_copy_source_session.cc:205] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 20:06:56.055850 27068 ts_tablet_manager.cc:1875] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 20:06:56.058032 27068 ts_tablet_manager.cc:1888] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 2.8
I20240501 20:06:56.058180 27068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca3285abe3654723a6922672391e9423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 20:06:56.059428 27068 tablet_copy_client.cc:783] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 5 data blocks...
W20240501 20:06:56.064445 14616 consensus_peers.cc:485] T e4b15761177f4cffb7aa5e4d16e42ac7 P c6d2c7096e404ea9a6f73b123d74c622 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:06:56.065722 27068 tablet_copy_client.cc:647] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Starting download of 1 WAL segments...
I20240501 20:06:56.067176 27068 tablet_copy_client.cc:523] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 20:06:56.068782 27068 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap starting.
I20240501 20:06:56.079615 27068 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 20:06:56.080190 27068 tablet_bootstrap.cc:492] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Bootstrap complete.
I20240501 20:06:56.080370 27068 ts_tablet_manager.cc:1367] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent bootstrapping tablet: real 0.012s user 0.012s sys 0.000s
I20240501 20:06:56.080816 27068 raft_consensus.cc:348] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 20:06:56.080947 27068 raft_consensus.cc:729] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 58f750e4d9574c9ebfd3d9e57eefd411, State: Initialized, Role: LEARNER
I20240501 20:06:56.081079 27068 consensus_queue.cc:260] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: NON_VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: true } }
I20240501 20:06:56.081396 27068 ts_tablet_manager.cc:1398] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 20:06:56.081869 13007 tablet_copy_service.cc:342] P 2d52575d9e1b40dca0fc553d7295eb74: Request end of tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-ca3285abe3654723a6922672391e9423 received from {username='slave'} at 127.5.139.66:43757
I20240501 20:06:56.081949 13007 tablet_copy_service.cc:433] P 2d52575d9e1b40dca0fc553d7295eb74: ending tablet copy session 58f750e4d9574c9ebfd3d9e57eefd411-ca3285abe3654723a6922672391e9423 on tablet ca3285abe3654723a6922672391e9423 with peer 58f750e4d9574c9ebfd3d9e57eefd411
I20240501 20:06:56.093533 26998 consensus_queue.cc:579] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.477s)
I20240501 20:06:56.096251 12725 consensus_queue.cc:237] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 1.17, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } }
I20240501 20:06:56.098137 12988 raft_consensus.cc:1261] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 17. Preceding OpId from leader: term: 1 index: 18. (index mismatch)
W20240501 20:06:56.098654 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:56.098693 26998 consensus_queue.cc:1035] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20240501 20:06:56.100693 26891 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: Committing config change with OpId 1.18: config changed from index 17 to 18, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 20:06:56.101262 12988 raft_consensus.cc:2932] T 8b61e94fc7e541c8aa84957560508a31 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.18: config changed from index 17 to 18, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } } }
I20240501 20:06:56.102308 12437 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8b61e94fc7e541c8aa84957560508a31 with cas_config_opid_index 17: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:56.103616 12445 catalog_manager.cc:5484] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 reported cstate change: config changed from index 17 to 18, NON_VOTER 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) added. New cstate: current_term: 1 leader_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: NON_VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 20:06:56.189231 12659 connection.cc:535] client connection to 127.5.139.69:34719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:06:56.189265 12528 connection.cc:535] client connection to 127.5.139.69:34719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:06:56.189280 12921 connection.cc:535] client connection to 127.5.139.69:34719 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:06:56.190380 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.190428 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.190485 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.190479 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.190523 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:56.190524 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.190636 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.198227 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:56.216840 12857 raft_consensus.cc:1203] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 20:06:56.219185 12773 maintenance_manager.cc:419] P 58f750e4d9574c9ebfd3d9e57eefd411: Scheduling UndoDeltaBlockGCOp(ca3285abe3654723a6922672391e9423): 4581 bytes on disk
I20240501 20:06:56.219630 27113 ts_tablet_manager.cc:902] T 32a17c2e392a49b7b181fa7a25a778a1 P c6d2c7096e404ea9a6f73b123d74c622: Initiating tablet copy from peer 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 20:06:56.219630 12663 maintenance_manager.cc:643] P 58f750e4d9574c9ebfd3d9e57eefd411: UndoDeltaBlockGCOp(ca3285abe3654723a6922672391e9423) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
F20240501 20:06:56.219821 27113 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.queue_time_us, tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.queue_time_us) (tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_cpu_time_us, tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_cpu_time_us) (tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_wall_time_us, tablet-download-pool-04ccc8addc55441b9ab9d8265b90b03b.run_wall_time_us) (tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.queue_time_us, tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.queue_time_us) (tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.run_cpu_time_us, tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.run_cpu_time_us) (tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.run_wall_time_us, tablet-download-pool-0bf64d9f3ce94db28b032cdcd795a255.run_wall_time_us) (tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.queue_time_us, tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.queue_time_us) (tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_cpu_time_us, tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_cpu_time_us) (tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_wall_time_us, tablet-download-pool-0fb481d394034a8f85af613beb5c9ec8.run_wall_time_us) (tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.queue_time_us, tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.queue_time_us) (tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.run_cpu_time_us, tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.run_cpu_time_us) (tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.run_wall_time_us, tablet-download-pool-22b272a8582d4060a6a527ac4fce9ae5.run_wall_time_us) (tablet-download-pool-32a17c2e392a49b7b181fa7a25a778a1.queue_time_us, tablet-download-pool-32a17c2e392a49b7b181fa7a25a778a1.queue_time_us) (tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.queue_time_us, tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.queue_time_us) (tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_cpu_time_us, tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_cpu_time_us) (tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_wall_time_us, tablet-download-pool-3e84a3865ce64eaebad4df46ec01842d.run_wall_time_us) (tablet-download-pool-544caa38514643d5b81cd6ed85e0bae7.queue_time_us, tablet-download-pool-544caa38514643d5b81cd6ed85e0bae7.queue_time_us) (tablet-download-pool-544caa38514643d5b81cd6ed85e0bae7.run_cpu_time_us, tablet-download-pool-544caa38514643d5b81cd6ed85e0bae7.run_cpu_time_us) (tablet-download-pool-544caa38514643d5b81cd6ed85e0bae7.run_wall_time_us, tablet-download-pool-544caa38514643d5b81cd6ed85e0bae7.run_wall_time_us) (tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.queue_time_us, tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.queue_time_us) (tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_cpu_time_us, tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_cpu_time_us) (tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_wall_time_us, tablet-download-pool-557994658e2c4c8bbc1eeacdab958ddd.run_wall_time_us) (tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.queue_time_us, tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.queue_time_us) (tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.run_cpu_time_us, tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.run_cpu_time_us) (tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.run_wall_time_us, tablet-download-pool-745e39ec23db4f90bb8b1e8b49a50244.run_wall_time_us) (tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.queue_time_us, tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.queue_time_us) (tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.run_cpu_time_us, tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.run_cpu_time_us) (tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.run_wall_time_us, tablet-download-pool-955edb7c12fa4b32ac58827f131931e3.run_wall_time_us) (tablet-download-pool-ce6add2f19a649b3a12497951047705c.queue_time_us, tablet-download-pool-ce6add2f19a649b3a12497951047705c.queue_time_us) (tablet-download-pool-ce6add2f19a649b3a12497951047705c.run_cpu_time_us, tablet-download-pool-ce6add2f19a649b3a12497951047705c.run_cpu_time_us) (tablet-download-pool-ce6add2f19a649b3a12497951047705c.run_wall_time_us, tablet-download-pool-ce6add2f19a649b3a12497951047705c.run_wall_time_us) (tablet-download-pool-d75bd8f836b947248ee124f456c7d721.queue_time_us, tablet-download-pool-d75bd8f836b947248ee124f456c7d721.queue_time_us) (tablet-download-pool-d75bd8f836b947248ee124f456c7d721.run_cpu_time_us, tablet-download-pool-d75bd8f836b947248ee124f456c7d721.run_cpu_time_us) (tablet-download-pool-d75bd8f836b947248ee124f456c7d721.run_wall_time_us, tablet-download-pool-d75bd8f836b947248ee124f456c7d721.run_wall_time_us) (tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.queue_time_us, tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.queue_time_us) (tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.run_cpu_time_us, tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.run_cpu_time_us) (tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.run_wall_time_us, tablet-download-pool-d940b16d07d54945abe4a10a11dd6a2a.run_wall_time_us) (tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.queue_time_us, tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.queue_time_us) (tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.run_cpu_time_us, tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.run_cpu_time_us) (tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.run_wall_time_us, tablet-download-pool-db240ec5f8b2444fb6114ad814f3cdb7.run_wall_time_us) (tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.queue_time_us, tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.queue_time_us) (tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_cpu_time_us, tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_cpu_time_us) (tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_wall_time_us, tablet-download-pool-e4b15761177f4cffb7aa5e4d16e42ac7.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f673f2b4dcd google::LogMessage::Fail() at ??:0
@ 0x7f673f2b8b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f673f2b47cc google::LogMessage::Flush() at ??:0
@ 0x7f673f2b5f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f674071e044 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f6740702f98 kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f6740700efc kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f67432b377a kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f67432bca45 kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f674333d76f kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 20:06:56.243248 12642 maintenance_manager.cc:419] P ebbbd9ac07a94e8dbbae6def0a2df415: Scheduling UndoDeltaBlockGCOp(800d5f43cd1b4fb6bd8bbcac262aeb80): 4920 bytes on disk
I20240501 20:06:56.243655 12532 maintenance_manager.cc:643] P ebbbd9ac07a94e8dbbae6def0a2df415: UndoDeltaBlockGCOp(800d5f43cd1b4fb6bd8bbcac262aeb80) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
@ 0x7f67433514f6 kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f674333c226 _ZZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS_6StatusENS0_24TabletServerErrorPB_CodeEEEENKUlvE_clEv at ??:0
@ 0x7f674334bc0a _ZNSt17_Function_handlerIFvvEZN4kudu7tserver15TSTabletManager15StartTabletCopyEPKNS1_9consensus24StartTabletCopyRequestPBESt8functionIFvRKNS1_6StatusENS2_24TabletServerErrorPB_CodeEEEEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55fd7033f75c std::function<>::operator()() at ??:0
@ 0x7f6740705e06 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f67407066ef _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f674070803b _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x55fd7033f75c std::function<>::operator()() at ??:0
@ 0x7f67406f5807 kudu::Thread::SuperviseThread() at ??:0
W20240501 20:06:56.262312 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f6740f336db start_thread at ??:0
@ 0x7f673e46b71f clone at ??:0
W20240501 20:06:56.292191 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:56.303146 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.375618 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:56.381676 27043 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:56.418998 12661 connection.cc:535] client connection to 127.5.139.72:37453 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:06:56.419000 12923 connection.cc:535] client connection to 127.5.139.72:37453 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:06:56.418992 12792 connection.cc:535] client connection to 127.5.139.72:37453 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 20:06:56.420120 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:06:56.420126 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.420215 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.420261 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.420282 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.434468 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.446475 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.466001 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:56.473846 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:56.485219 12725 raft_consensus.cc:1203] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20240501 20:06:56.493149 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.504817 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.505417 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:56.540032 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:56.563880 12725 raft_consensus.cc:1203] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
W20240501 20:06:56.565784 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.609820 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:56.629554 12592 raft_consensus.cc:1203] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240501 20:06:56.669118 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:56.679281 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:06:56.729871 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:56.741075 27021 raft_consensus.cc:1050] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415: attempting to promote NON_VOTER a238fc0c6d844cedb8adccb97d01ae5e to VOTER
I20240501 20:06:56.741465 27021 consensus_queue.cc:237] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false } }
I20240501 20:06:56.743069 12857 raft_consensus.cc:1261] T d940b16d07d54945abe4a10a11dd6a2a P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEARNER]: Refusing update from remote peer ebbbd9ac07a94e8dbbae6def0a2df415: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240501 20:06:56.743520 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.743868 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:06:56.743867 27021 consensus_queue.cc:1035] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:06:56.743978 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:56.776083 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:06:56.798969 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:56.878119 27043 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:56.918977 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:56.976084 27149 raft_consensus.cc:1050] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e: attempting to promote NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 to VOTER
I20240501 20:06:56.976459 27149 consensus_queue.cc:237] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:06:56.977962 12725 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEARNER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:06:56.978138 12988 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 20:06:56.978592 27148 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 20:06:56.978780 12793 consensus_peers.cc:485] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:56.978791 27165 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 20:06:56.980401 27148 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:56.980847 12988 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:56.982406 12725 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:56.983525 12445 catalog_manager.cc:5484] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 9 to 10, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:06:56.987280 12857 consensus_queue.cc:237] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:06:56.988204 12725 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:06:56.988296 12988 raft_consensus.cc:1261] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:06:56.988545 27013 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:06:56.988679 27149 consensus_queue.cc:1035] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:06:56.990142 27148 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:56.990572 12988 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 2d52575d9e1b40dca0fc553d7295eb74 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:56.990559 12725 raft_consensus.cc:2932] T 04a70a2ee2144f2287795ffe723d2716 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:56.991688 12437 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 04a70a2ee2144f2287795ffe723d2716 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:06:56.992774 12449 catalog_manager.cc:5484] T 04a70a2ee2144f2287795ffe723d2716 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: config changed from index 10 to 11, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New cstate: current_term: 1 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:06:57.069703 27043 raft_consensus.cc:482] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9069ec6fcdb248479037f2d37e0e851d)
I20240501 20:06:57.069842 27043 raft_consensus.cc:504] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:06:57.070309 27043 leader_election.cc:288] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719)
I20240501 20:06:57.070722 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d" candidate_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
W20240501 20:06:57.071347 12659 leader_election.cc:334] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:57.071410 12659 leader_election.cc:302] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411; no voters: 9069ec6fcdb248479037f2d37e0e851d, a238fc0c6d844cedb8adccb97d01ae5e
I20240501 20:06:57.071537 27043 raft_consensus.cc:2726] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:06:57.074084 27140 raft_consensus.cc:1050] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74: attempting to promote NON_VOTER 58f750e4d9574c9ebfd3d9e57eefd411 to VOTER
I20240501 20:06:57.074473 27140 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:06:57.075711 12725 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 LEARNER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:06:57.075877 12857 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240501 20:06:57.076273 27172 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 20:06:57.076515 27140 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:06:57.076956 12924 consensus_peers.cc:485] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:57.078812 27172 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:57.079222 12857 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:57.081105 27140 consensus_queue.cc:579] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.291s)
I20240501 20:06:57.081718 12451 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 10 to 11, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:06:57.082481 12725 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:57.085582 12988 consensus_queue.cc:237] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:06:57.087143 12857 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:06:57.087148 12725 raft_consensus.cc:1261] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 20:06:57.087500 27172 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:06:57.087599 27140 consensus_queue.cc:1035] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 20:06:57.088971 27041 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:57.089380 12725 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:57.090389 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ca3285abe3654723a6922672391e9423 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:06:57.090513 12857 raft_consensus.cc:2932] T ca3285abe3654723a6922672391e9423 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } } }
I20240501 20:06:57.091298 12449 catalog_manager.cc:5484] T ca3285abe3654723a6922672391e9423 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 11 to 12, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:06:57.133416 27041 consensus_queue.cc:579] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.002s)
I20240501 20:06:57.134532 27172 consensus_queue.cc:579] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Leader has been unable to successfully communicate with peer 2ca06dad25a645bca71b09741798194a for more than 300 seconds (300.344s)
W20240501 20:06:57.135512 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:06:57.157076 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:57.175419 27043 raft_consensus.cc:1050] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411: attempting to promote NON_VOTER ebbbd9ac07a94e8dbbae6def0a2df415 to VOTER
I20240501 20:06:57.175810 27043 consensus_queue.cc:237] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } }
I20240501 20:06:57.177243 12592 raft_consensus.cc:1261] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 1 LEARNER]: Refusing update from remote peer 58f750e4d9574c9ebfd3d9e57eefd411: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 20:06:57.177678 27181 consensus_queue.cc:1035] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 20:06:57.178035 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:57.178151 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:57.199724 12924 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 20:06:57.253866 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:06:57.348666 27172 raft_consensus.cc:482] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9069ec6fcdb248479037f2d37e0e851d)
I20240501 20:06:57.348793 27172 raft_consensus.cc:504] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:06:57.349224 27172 leader_election.cc:288] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67:46467), 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719)
I20240501 20:06:57.349675 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "2d52575d9e1b40dca0fc553d7295eb74" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" is_pre_election: true
W20240501 20:06:57.350272 12921 leader_election.cc:334] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:57.350410 12921 leader_election.cc:302] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, a238fc0c6d844cedb8adccb97d01ae5e
I20240501 20:06:57.350580 27172 raft_consensus.cc:2726] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:06:57.354203 27013 raft_consensus.cc:482] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9069ec6fcdb248479037f2d37e0e851d)
I20240501 20:06:57.354313 27013 raft_consensus.cc:504] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:06:57.354698 27013 leader_election.cc:288] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
W20240501 20:06:57.355139 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 20:06:57.355193 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" is_pre_election: true
I20240501 20:06:57.355336 12725 raft_consensus.cc:2454] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 2.
I20240501 20:06:57.355669 12790 leader_election.cc:302] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
W20240501 20:06:57.355814 12790 leader_election.cc:334] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:57.355841 27013 raft_consensus.cc:2781] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:06:57.355890 27013 raft_consensus.cc:482] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9069ec6fcdb248479037f2d37e0e851d)
I20240501 20:06:57.355942 27013 raft_consensus.cc:3037] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:06:57.356797 27013 raft_consensus.cc:504] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:06:57.356871 27172 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:06:57.356943 27172 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:06:57.357161 27013 leader_election.cc:288] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 election: Requested vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895)
I20240501 20:06:57.357290 27172 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
I20240501 20:06:57.357661 12725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e84a3865ce64eaebad4df46ec01842d" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 3 candidate_status { last_received { term: 2 index: 16 } } ignore_live_leader: false dest_uuid: "58f750e4d9574c9ebfd3d9e57eefd411"
I20240501 20:06:57.357789 12725 raft_consensus.cc:3037] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:06:57.358285 12790 leader_election.cc:334] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:57.358698 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:06:57.358846 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:57.358878 12725 raft_consensus.cc:2454] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 3.
I20240501 20:06:57.358901 12921 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:06:57.359127 27172 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:06:57.359158 12790 leader_election.cc:302] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 58f750e4d9574c9ebfd3d9e57eefd411, a238fc0c6d844cedb8adccb97d01ae5e; no voters: 9069ec6fcdb248479037f2d37e0e851d
I20240501 20:06:57.359316 27013 raft_consensus.cc:2781] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:06:57.359405 27013 raft_consensus.cc:686] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [term 3 LEADER]: Becoming Leader. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Running, Role: LEADER
I20240501 20:06:57.359546 27013 consensus_queue.cc:237] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 16, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } }
I20240501 20:06:57.361428 12449 catalog_manager.cc:5484] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: term changed from 2 to 3, leader changed from 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) to a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67). New cstate: current_term: 3 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 20:06:57.385514 26891 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:57.392127 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:57.395488 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:57.419646 27149 raft_consensus.cc:482] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 9069ec6fcdb248479037f2d37e0e851d)
I20240501 20:06:57.419785 27149 raft_consensus.cc:504] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:06:57.420193 27149 leader_election.cc:288] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 20:06:57.420653 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74" is_pre_election: true
I20240501 20:06:57.420838 12988 raft_consensus.cc:2454] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 2.
I20240501 20:06:57.421175 12793 leader_election.cc:302] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters:
W20240501 20:06:57.421267 12790 leader_election.cc:334] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:57.421343 27149 raft_consensus.cc:2781] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 20:06:57.421396 27149 raft_consensus.cc:482] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 9069ec6fcdb248479037f2d37e0e851d)
I20240501 20:06:57.421443 27149 raft_consensus.cc:3037] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 2 FOLLOWER]: Advancing to term 3
I20240501 20:06:57.422312 27149 raft_consensus.cc:504] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:06:57.422643 27149 leader_election.cc:288] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 election: Requested vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), 2d52575d9e1b40dca0fc553d7295eb74 (127.5.139.68:41997)
I20240501 20:06:57.423009 12988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf6948589837482a9f580731e3b861b8" candidate_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "2d52575d9e1b40dca0fc553d7295eb74"
I20240501 20:06:57.423116 12988 raft_consensus.cc:3037] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Advancing to term 3
W20240501 20:06:57.423511 12790 leader_election.cc:334] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:57.424068 12988 raft_consensus.cc:2454] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a238fc0c6d844cedb8adccb97d01ae5e in term 3.
I20240501 20:06:57.424343 12793 leader_election.cc:302] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74, a238fc0c6d844cedb8adccb97d01ae5e; no voters: 9069ec6fcdb248479037f2d37e0e851d
I20240501 20:06:57.424515 27149 raft_consensus.cc:2781] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 20:06:57.424607 27149 raft_consensus.cc:686] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [term 3 LEADER]: Becoming Leader. State: Replica: a238fc0c6d844cedb8adccb97d01ae5e, State: Running, Role: LEADER
I20240501 20:06:57.424774 27149 consensus_queue.cc:237] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:06:57.426561 12449 catalog_manager.cc:5484] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e reported cstate change: term changed from 2 to 3, leader changed from 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69) to a238fc0c6d844cedb8adccb97d01ae5e (127.5.139.67). New cstate: current_term: 3 leader_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "a238fc0c6d844cedb8adccb97d01ae5e" member_type: VOTER last_known_addr { host: "127.5.139.67" port: 46467 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 20:06:57.586194 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 04a70a2ee2144f2287795ffe723d2716: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:06:57.670487 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:06:57.734323 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 20:06:57.778049 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:57.803794 12986 raft_consensus.cc:1261] T cf6948589837482a9f580731e3b861b8 P 2d52575d9e1b40dca0fc553d7295eb74 [term 3 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20240501 20:06:57.804210 27149 consensus_queue.cc:1035] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 20:06:57.807808 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 20:06:57.853029 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:57.887809 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:57.897917 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:57.926267 12725 raft_consensus.cc:1261] T 3e84a3865ce64eaebad4df46ec01842d P 58f750e4d9574c9ebfd3d9e57eefd411 [term 3 FOLLOWER]: Refusing update from remote peer a238fc0c6d844cedb8adccb97d01ae5e: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 3 index: 17. (index mismatch)
I20240501 20:06:57.926734 27203 consensus_queue.cc:1035] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 20:06:57.964668 27181 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:57.993893 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:58.064743 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:06:58.224367 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.375249 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:58.398983 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 20:06:58.399646 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:58.401363 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:58.417179 27181 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:58.452445 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:06:58.467254 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.508898 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.520308 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.554793 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 20:06:58.567797 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.581359 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.614058 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.643199 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:06:58.660766 12442 maintenance_manager.cc:419] P be1305e487d64b748456a721443e3b3b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2025 bytes on disk
I20240501 20:06:58.663050 12441 maintenance_manager.cc:643] P be1305e487d64b748456a721443e3b3b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.002s sys 0.000s Metrics: {"cfile_init":1,"dirs.queue_time_us":132,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"thread_start_us":133,"threads_started":2,"wal-append.queue_time_us":159}
I20240501 20:06:58.689986 27229 consensus_queue.cc:579] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Leader has been unable to successfully communicate with peer 1d49e80be3d04fb99b711c62d61bac09 for more than 300 seconds (300.110s)
I20240501 20:06:58.693122 12988 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 2.15, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } }
I20240501 20:06:58.694463 12592 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240501 20:06:58.694463 12725 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 15. Preceding OpId from leader: term: 2 index: 16. (index mismatch)
I20240501 20:06:58.694840 27172 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 20:06:58.694972 27232 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20240501 20:06:58.696856 27229 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:06:58.697397 12592 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:06:58.698643 12725 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.16: config changed from index 15 to 16, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New config: { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } }
I20240501 20:06:58.700449 12440 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8bc9fe9ade8246779164d6db31b11e16 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 20:06:58.701097 12449 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 reported cstate change: config changed from index 15 to 16, VOTER 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71) evicted. New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 20:06:58.704712 12988 consensus_queue.cc:237] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 2.16, Last appended by leader: 11, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } }
W20240501 20:06:58.704777 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:58.706034 12592 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240501 20:06:58.706045 12725 raft_consensus.cc:1261] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Refusing update from remote peer 2d52575d9e1b40dca0fc553d7295eb74: Log matching property violated. Preceding OpId in replica: term: 2 index: 16. Preceding OpId from leader: term: 2 index: 17. (index mismatch)
I20240501 20:06:58.706396 27229 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20240501 20:06:58.706849 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 20:06:58.707115 27229 consensus_queue.cc:1035] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20240501 20:06:58.708137 27232 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 LEADER]: Committing config change with OpId 2.17: config changed from index 16 to 17, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 20:06:58.708576 12592 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 20:06:58.709115 12725 raft_consensus.cc:2932] T 8bc9fe9ade8246779164d6db31b11e16 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 2 FOLLOWER]: Committing config change with OpId 2.17: config changed from index 16 to 17, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
I20240501 20:06:58.711015 12440 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8bc9fe9ade8246779164d6db31b11e16 with cas_config_opid_index 16: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 20:06:58.711198 12449 catalog_manager.cc:5484] T 8bc9fe9ade8246779164d6db31b11e16 P ebbbd9ac07a94e8dbbae6def0a2df415 reported cstate change: config changed from index 16 to 17, NON_VOTER c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72) added. New cstate: current_term: 2 leader_uuid: "2d52575d9e1b40dca0fc553d7295eb74" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } attrs { replace: true } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } } peers { permanent_uuid: "ebbbd9ac07a94e8dbbae6def0a2df415" member_type: VOTER last_known_addr { host: "127.5.139.65" port: 38745 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: NON_VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: true } } }
W20240501 20:06:58.727283 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.731870 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.762454 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.767937 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.797211 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:58.846580 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.870783 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 20:06:58.870877 27181 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:58.925534 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.935283 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:58.937821 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:06:58.963440 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:58.970103 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.983462 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:58.994114 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:59.078122 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:59.085373 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:59.094919 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:59.130036 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 20:06:59.147732 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:59.164729 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 20:06:59.171126 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:59.207221 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 99 similar messages]
W20240501 20:06:59.279079 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:59.414950 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:06:59.422590 27262 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:06:59.463172 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:06:59.475178 27229 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:06:59.475314 27229 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:06:59.475749 27229 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:06:59.476670 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:59.476761 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:06:59.476841 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:06:59.476966 27229 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:06:59.495265 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:59.511536 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 20:06:59.514891 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:59.727909 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:59.731757 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 20:06:59.785548 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:06:59.820505 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:59.885689 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:06:59.901226 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:06:59.904306 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
I20240501 20:06:59.934125 27276 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:00.038159 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 20:07:00.135047 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:00.149816 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:07:00.243352 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:00.271962 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:07:00.350637 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:00.479422 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:00.519762 27276 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:00.551415 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:00.673079 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:00.711244 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:00.770247 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:00.815331 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:00.873502 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:00.901984 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:00.931645 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:00.940526 12440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 31 similar messages]
W20240501 20:07:00.941231 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:00.945768 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:07:01.012228 27276 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:01.027033 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.040517 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.049242 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:01.058509 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.110154 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.132994 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.159955 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:01.193189 27229 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:01.193352 27229 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:01.193748 27229 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:01.193946 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.194634 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:01.194808 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:01.194880 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:01.195004 27229 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:07:01.218045 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.223163 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.240394 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.270403 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 20:07:01.295544 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:01.348778 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:01.411144 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.426260 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.431691 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:07:01.467823 27295 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:01.546169 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:01.549523 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 20:07:01.553354 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:01.557325 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.560653 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.600150 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.629478 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.629884 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 606: this message will repeat every 5th retry.
W20240501 20:07:01.660400 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.675487 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.749349 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:01.779345 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:01.845727 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 20:07:01.854862 27276 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:01.938663 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:02.059721 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:02.099843 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:02.168890 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 20:07:02.282701 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 20:07:02.366297 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:02.370579 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:02.382524 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
I20240501 20:07:02.413534 27295 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:02.475318 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:02.500303 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:02.504953 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:07:02.526122 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:02.663350 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:02.672730 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:02.783289 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 20:07:02.832491 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:03.015686 27295 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:03.037981 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:03.065938 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:03.089864 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.120108 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:03.193481 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.225847 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.257992 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.282838 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:03.417675 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 20:07:03.422880 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.449134 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:03.455463 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 20:07:03.463867 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.532469 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:03.592871 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.608543 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:07:03.623875 27385 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:03.625953 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 20:07:03.673985 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:03.692314 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:03.693497 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.707485 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.744359 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.748040 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.753716 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.818490 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:03.851485 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.891000 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.898370 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:03.976362 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:03.989814 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.029529 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.063863 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 20:07:04.069114 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:04.114331 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:04.147612 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.153584 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.179484 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.195946 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:04.199190 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.208134 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:07:04.234548 27401 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:04.245360 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.299968 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:07:04.316220 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.423525 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:04.441215 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 611: this message will repeat every 5th retry.
W20240501 20:07:04.533746 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:04.595484 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:04.615326 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 20:07:04.640945 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:04.735180 27229 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:04.735314 27229 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:04.735733 27229 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:04.736795 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:04.736883 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:04.736990 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:04.737146 27229 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:07:04.793972 27401 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:04.828055 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.831223 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 20:07:04.842499 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
W20240501 20:07:04.871879 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:04.909400 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:04.995764 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:05.026197 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:05.036780 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:05.041405 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 20:07:05.355518 27401 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:05.394577 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 04a70a2ee2144f2287795ffe723d2716: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:05.466377 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:05.478287 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:05.569172 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:05.583963 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:05.585129 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:05.636382 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:05.688315 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:05.793222 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:05.798099 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 20:07:05.814066 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:05.840380 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:07:05.859548 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:05.939178 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:05.940707 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:06.036280 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:06.051018 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:06.099678 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.161697 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.161674 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:06.166853 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.189481 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.195622 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.214951 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.243814 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.333319 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 20:07:06.385821 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:06.431031 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.503875 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:06.506325 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:06.508157 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:06.565346 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.577778 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:06.595635 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.672394 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.693914 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.700210 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 20:07:06.767175 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.771148 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.816361 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 616: this message will repeat every 5th retry.
I20240501 20:07:06.827515 27401 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:07:06.872633 27449 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:06.872749 27449 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:06.873111 27449 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:06.874156 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:06.874243 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:06.874336 12921 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:06.874524 27449 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:07:06.877317 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.878376 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:06.887977 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:06.892108 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 20:07:06.913067 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:07.005674 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:07.033702 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:07.034510 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:07.057842 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 20:07:07.088461 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:07.118127 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:07.154999 12440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240501 20:07:07.155773 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:07.248674 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
I20240501 20:07:07.336659 27401 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:07.467397 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:07.486485 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:07.500737 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:07.512506 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:07.613651 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:07.753003 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:07.758345 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:07.796917 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 20:07:07.917567 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:07.947432 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:08.009544 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:08.026480 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.048985 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:08.147342 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:08.158123 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.204775 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.262748 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 20:07:08.289952 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.298650 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:08.332401 27401 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:08.451447 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:07:08.452167 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:08.514024 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.516671 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:08.544539 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:08.567173 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.591408 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.600045 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.636816 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.715571 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.718845 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 20:07:08.719581 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.721221 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.785789 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:07:08.879997 27401 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:08.883677 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.889400 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:08.918437 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:08.918798 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:09.038771 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.099766 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:09.106400 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.133529 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.135921 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:09.138705 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 621: this message will repeat every 5th retry.
W20240501 20:07:09.189157 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.196187 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.249830 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.323262 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:07:09.323933 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 20:07:09.354532 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.408744 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.424376 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:09.468008 27401 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:09.472602 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.503468 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:09.517251 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 20:07:09.537106 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:09.551434 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.586179 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:09.602450 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:09.682896 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 20:07:09.825351 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:09.868094 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 20:07:09.868924 27401 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:09.981230 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:10.050757 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:10.078473 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:10.132678 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
I20240501 20:07:10.313704 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:10.327731 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:10.369446 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:10.431217 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:10.509759 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:10.522603 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:10.612967 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:10.650116 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:10.653682 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:10.810492 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:07:10.861013 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:10.878484 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:10.897320 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:10.905872 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:10.925964 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:10.968174 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:10.974259 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:10.994505 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:11.054046 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 20:07:11.117120 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.121800 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.172487 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:11.234454 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.276518 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:07:11.338229 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:11.346051 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.399117 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.408011 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.444859 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:11.469733 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:11.479341 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:11.486228 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 626: this message will repeat every 5th retry.
W20240501 20:07:11.526178 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.542029 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.641253 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.647274 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.685758 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.775732 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.780895 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.797226 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:11.803539 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.852777 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.860611 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 20:07:11.883589 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:11.897668 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 20:07:11.928017 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 20:07:11.940794 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:11.946583 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.963529 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:11.988050 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:12.019429 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 20:07:12.050410 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:12.103037 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:12.282646 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:12.317015 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:12.402926 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:12.570081 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:12.575559 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:12.599375 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:12.649096 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
W20240501 20:07:12.828254 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:12.847584 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 20:07:12.857224 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:12.881157 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:12.890682 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:13.012203 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:13.043217 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:13.139919 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.173980 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:13.190913 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:13.301561 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:13.312875 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.313341 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.370556 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:13.426296 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.444905 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:07:13.491353 27659 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:13.508625 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.521149 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 20:07:13.521874 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.574132 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 20:07:13.607055 12440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 20:07:13.607825 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 04a70a2ee2144f2287795ffe723d2716: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:13.644447 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.665606 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:13.739346 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 106 similar messages]
W20240501 20:07:13.745522 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.778638 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:13.781764 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.812098 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:13.850700 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:07:13.897234 27659 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:13.922858 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:13.924907 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.931187 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:13.962517 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 631: this message will repeat every 5th retry.
I20240501 20:07:14.075946 27647 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:14.076076 27647 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:14.076462 27647 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:14.077407 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:14.077502 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:14.077598 12921 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:14.077773 27647 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:07:14.084580 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.101722 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.125514 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.192849 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.195417 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 20:07:14.203554 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:14.213061 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.262756 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:14.299868 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.322070 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.339568 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.386952 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 20:07:14.399412 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 20:07:14.402498 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:14.439965 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:14.451421 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.471673 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.487370 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:14.490615 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.503284 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:14.757234 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:14.801101 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 20:07:14.844622 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:14.862465 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:14.938776 27659 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:14.955436 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:15.144438 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:15.157337 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 20:07:15.273312 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:15.366667 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:15.383147 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 20:07:15.391256 27451 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:15.415666 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:15.419592 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:15.431528 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:15.528560 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:15.531111 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:15.690732 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:15.768518 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:15.815169 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:15.836998 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:15.857111 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:15.905010 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:15.909197 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:15.929769 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:07:15.955660 27727 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:16.001681 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.008332 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.008414 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.035126 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.075263 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 20:07:16.089202 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.131609 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:16.226958 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.289132 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:16.345139 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:07:16.368424 27727 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:16.376214 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:16.409940 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:16.412127 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.447510 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.471908 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.511375 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.511700 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.561316 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 20:07:16.638427 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.654899 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.683785 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:16.712365 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 636: this message will repeat every 5th retry.
W20240501 20:07:16.720853 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:16.753365 27647 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:16.753494 27647 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:16.753890 27647 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:16.754858 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:16.754971 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:16.755055 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:16.755184 27647 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:07:16.756001 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.783308 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.842399 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.855126 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
I20240501 20:07:16.859454 27727 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:16.887115 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.895543 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.910202 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:16.978430 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:16.990403 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:17.071352 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:17.101755 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:17.145928 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 20:07:17.246440 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:17.327607 27727 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:17.373247 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:17.418113 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:17.546422 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:17.632210 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:17.753948 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:17.766263 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:07:17.777030 27727 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:17.879163 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 20:07:17.879724 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 20:07:17.881844 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:17.899289 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:18.080672 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:18.085242 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.086541 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:18.105451 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 20:07:18.297567 27727 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:18.333748 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.343842 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.364483 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:18.367149 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:18.394032 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.434299 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.437857 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.463102 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.470289 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.481814 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:18.488642 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.578660 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.630859 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:18.635272 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 20:07:18.682297 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.727373 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 20:07:18.730101 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.830945 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 20:07:18.834715 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:18.858717 27790 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:18.872701 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:18.885275 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:18.909715 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.046567 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.064805 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 20:07:19.074755 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:19.088740 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:19.093261 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.103031 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.133030 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.148121 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.190690 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 641: this message will repeat every 5th retry.
W20240501 20:07:19.199520 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.265370 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.276357 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.282662 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.290148 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:19.347015 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 20:07:19.375414 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:07:19.387521 27819 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:19.396834 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:07:19.496538 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:19.603286 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:19.661118 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:19.672473 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:19.705591 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 20:07:19.785270 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:19.820564 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 20:07:19.889393 27819 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:19.910316 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:20.031857 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:20.032750 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 20:07:20.128358 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:20.234525 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:20.250444 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:20.304406 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:20.327189 27836 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:20.407449 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:20.594795 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:20.604246 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:20.650187 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:20.738030 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:20.758095 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:07:20.763461 27819 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:20.774024 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:20.776859 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:20.781332 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:20.795776 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:20.869210 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:20.941370 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.052115 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.063356 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.124083 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:21.138054 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.144505 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.201007 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.245611 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:21.245988 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.274781 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:21.282970 27836 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:21.431574 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:21.445600 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.479645 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.479863 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.558138 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.570380 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.591437 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 20:07:21.605468 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.611742 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:21.619386 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 20:07:21.636018 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:21.680637 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.698777 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 646: this message will repeat every 5th retry.
W20240501 20:07:21.701970 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.750869 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.750877 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
I20240501 20:07:21.769001 27819 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:21.778388 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.854472 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:21.869338 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:21.875531 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:21.998978 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:22.018846 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:22.138829 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:22.218624 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:22.230684 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 20:07:22.272745 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:22.323486 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:22.344601 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 20:07:22.350275 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 20:07:22.384615 27836 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:22.482143 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:22.493299 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:22.831947 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:22.832419 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:22.850922 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:22.882654 27836 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:22.987303 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.040613 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:23.061247 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:23.061822 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 20:07:23.062435 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.075872 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.129472 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.146409 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.216392 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.222632 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:23.231170 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:23.313170 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:23.332954 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:23.434967 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:07:23.436566 27819 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:23.538794 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.555655 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.582615 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.602722 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:23.647497 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.657230 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 20:07:23.697441 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:23.731372 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.812497 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:23.885955 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.897511 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:07:23.914893 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:23.977275 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:23.994678 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:24.001215 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:24.028398 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:24.033232 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:24.036671 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:07:24.043278 27946 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:24.055951 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:24.091962 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 651: this message will repeat every 5th retry.
W20240501 20:07:24.163344 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 20:07:24.201287 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:24.217403 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 20:07:24.244307 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:24.254658 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 20:07:24.270995 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:24.300309 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:24.301237 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:24.376188 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:24.411936 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:07:24.434263 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:24.569934 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:24.591428 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:07:24.649942 27946 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:24.661104 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:24.827546 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:24.859606 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 20:07:24.888916 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:24.890134 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:24.914976 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:25.047389 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 20:07:25.062124 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:25.088958 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:25.165624 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 20:07:25.243968 27953 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:25.244107 27953 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:25.244555 27953 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:25.245579 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:25.245682 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:25.245759 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:25.245898 27953 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:07:25.267452 27946 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:25.379118 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:25.437939 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:25.440865 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:25.480448 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:25.491493 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:25.505537 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:25.509577 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:25.558727 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:25.685412 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:25.747637 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:07:25.804783 27946 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:25.837162 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:25.846021 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:25.892532 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:25.896847 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:25.929595 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:25.991086 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.071765 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:26.092206 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:26.095450 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.102171 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.135695 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.146123 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
I20240501 20:07:26.186054 27997 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:26.249641 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:26.255923 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.361954 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.394445 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.423611 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.424443 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.451640 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:26.474792 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.498203 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:26.498236 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 656: this message will repeat every 5th retry.
W20240501 20:07:26.596217 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:26.688112 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.731869 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.752307 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:26.755946 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240501 20:07:26.778318 27946 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:26.830807 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.859508 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.895495 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 20:07:26.900735 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:26.976133 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:27.015676 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:27.051609 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:27.067346 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 20:07:27.093794 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:27.127560 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:27.289110 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:27.352564 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:27.353415 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
I20240501 20:07:27.360205 28043 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:27.440733 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:27.444058 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:27.520375 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 20:07:27.551121 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:27.575044 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:27.666512 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:27.719704 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:07:27.941466 28059 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:27.968062 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:27.994947 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:28.005952 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.035377 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:28.104076 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:28.118620 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 20:07:28.119192 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.189702 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:28.214923 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:28.249507 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.277016 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.395534 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.405799 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.481801 12440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240501 20:07:28.482534 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:28.498698 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:07:28.519573 28059 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:28.531618 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:28.590291 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:28.620988 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.621862 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:28.665371 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.687474 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:28.692111 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.699949 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.705535 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.789779 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 20:07:28.800065 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:28.961269 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 111 similar messages]
I20240501 20:07:28.974054 28043 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:29.011566 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:29.064692 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.065016 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:29.086544 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.090114 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.106866 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.106979 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 661: this message will repeat every 5th retry.
W20240501 20:07:29.108173 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.116596 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:29.128072 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.210060 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.233650 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:29.245348 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 20:07:29.381023 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 20:07:29.421866 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.507565 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:07:29.508235 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 20:07:29.523815 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:29.529292 28101 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:29.530097 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:29.578338 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.610757 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:29.662315 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.690212 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.737344 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 20:07:29.840025 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:29.912957 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:29.922658 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:29.936471 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:29.937363 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:30.015190 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:30.017694 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 04a70a2ee2144f2287795ffe723d2716: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:30.057391 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:30.071223 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
I20240501 20:07:30.129890 28121 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:30.249619 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:30.332875 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:30.389009 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:30.450510 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:07:30.544391 28121 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:30.555508 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:30.608541 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:30.648597 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:30.737293 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:30.779330 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:30.794772 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:30.832643 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:30.878981 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:30.909435 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:30.963176 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:30.969319 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.009423 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
I20240501 20:07:31.048278 28147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:31.065861 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.152901 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.178400 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:31.192592 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:31.279145 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:31.290730 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.294266 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:31.320508 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.323163 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.332751 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.359961 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:07:31.519078 28147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:31.528975 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.539053 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 666: this message will repeat every 5th retry.
W20240501 20:07:31.581737 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.586486 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.625136 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:31.666385 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 20:07:31.670120 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:31.674733 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:07:31.676023 28099 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:31.676117 28099 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:31.676494 28099 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:31.677357 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:31.678014 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:31.678089 12921 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:31.678227 28099 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:07:31.696722 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:07:31.729434 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:31.770422 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.772591 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:07:31.775439 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:31.819432 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 20:07:31.858449 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:31.890738 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 20:07:32.017385 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:32.026233 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:32.049211 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
I20240501 20:07:32.118738 28160 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:32.182358 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 20:07:32.271088 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:32.305414 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:32.314451 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:32.352586 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:32.408002 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:32.411971 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:32.452971 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:32.453349 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:32.455240 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:32.551606 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
I20240501 20:07:32.576294 28160 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:32.839951 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:32.855319 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:32.893998 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:32.936350 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:32.936722 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:32.981115 28147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:33.122247 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:07:33.122854 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.197868 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.272866 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.306927 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:33.323666 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.325209 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:33.335341 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:33.367913 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:33.384292 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.440969 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:33.461933 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:33.538043 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:07:33.538453 28217 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:33.575582 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.616487 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.682142 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.743011 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.768106 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:33.781860 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.827657 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 20:07:33.828334 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:33.856628 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:33.936179 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:33.939635 28217 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:33.964730 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 20:07:33.965443 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.030248 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.050500 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:34.058010 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.098716 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.105652 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.201804 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:34.226420 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.264729 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 20:07:34.277804 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 671: this message will repeat every 5th retry.
W20240501 20:07:34.291759 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 20:07:34.311340 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:34.319135 28147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:34.397408 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.401752 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:34.423434 28099 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:34.423573 28099 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:34.423956 28099 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:34.424934 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:34.425029 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:34.425123 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:34.425251 28099 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:07:34.464221 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.497072 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:34.569638 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.583333 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:34.610004 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:07:34.614384 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 20:07:34.706578 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 20:07:34.776053 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:34.792083 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:34.795432 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.871819 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:07:34.897589 28251 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:34.953104 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.958110 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:34.997413 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:35.011973 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:35.077522 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 20:07:35.126185 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:35.227486 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:35.353113 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:35.356505 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:07:35.481263 28147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:35.564993 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:35.634232 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:35.677206 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:35.691816 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:35.815050 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:35.851574 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:35.876180 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:35.934381 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 20:07:35.944684 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:35.954417 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:07:35.966928 28147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:35.975020 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.077364 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.117820 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.119722 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:36.133473 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:36.206420 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.301914 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.338732 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.382957 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.407508 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:36.432044 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:36.482252 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:07:36.499845 28296 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:36.509701 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.530304 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:36.548705 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.600899 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 20:07:36.650158 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.669435 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:36.690459 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.693992 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.730640 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.828956 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.839900 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240501 20:07:36.879436 28296 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:36.891423 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.899811 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 676: this message will repeat every 5th retry.
W20240501 20:07:36.965026 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:36.968127 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:36.994879 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:37.080580 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:37.105736 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:37.129318 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 20:07:37.179981 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:07:37.314661 28147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:37.396860 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:37.424443 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:37.461702 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 20:07:37.560446 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:37.599457 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:37.600363 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:37.608259 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:37.610059 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 20:07:37.620249 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:37.702984 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:37.730417 28147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:37.775599 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:37.856804 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:37.951068 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:38.001767 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:38.121868 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.131763 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 20:07:38.138737 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.233011 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:38.254570 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.293581 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 20:07:38.302842 28345 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:38.359889 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:38.405306 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.480631 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:38.487392 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:38.541740 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.562014 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.579136 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:38.600910 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.640290 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.701719 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.728171 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.791896 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:38.811945 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:38.834246 28362 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:38.838609 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:07:38.860739 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:38.948765 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:38.949544 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 20:07:39.009711 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 20:07:39.010345 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.028688 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.043320 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.047545 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.109567 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:39.123337 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 20:07:39.124800 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.157404 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.268975 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:39.289448 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.291528 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:39.294893 28345 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:39.304965 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.372061 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.458617 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:39.479837 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.596594 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:39.596660 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:39.634320 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:07:39.634953 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 20:07:39.661404 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:39.755990 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:07:39.769897 28345 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:39.781452 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 20:07:39.814445 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 681: this message will repeat every 5th retry.
W20240501 20:07:39.856081 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:39.920347 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 20:07:40.007588 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:40.048071 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:40.057065 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:40.181355 28362 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:40.260453 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:40.302845 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:40.326717 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:40.431268 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:40.443876 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:40.452256 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:40.467242 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:40.484102 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:40.557372 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:40.559214 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:40.699124 28413 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:40.744915 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:40.747495 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:40.847281 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:40.948601 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:40.954110 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:40.982415 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:40.997879 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:41.026906 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.090092 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:07:41.117887 28413 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:41.152123 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.212818 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.267343 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 20:07:41.359926 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:41.362520 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.377918 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:41.392589 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.395128 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.489223 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:41.522238 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.546195 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:41.547343 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 20:07:41.551023 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.551578 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.553333 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.561143 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.562394 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
I20240501 20:07:41.621455 28413 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:41.887550 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.898061 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:41.927524 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:41.927985 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:42.017377 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:42.055557 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:42.060179 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:42.071707 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 20:07:42.074287 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:42.112659 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 20:07:42.117969 28362 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:42.321530 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:42.345983 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:42.394385 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 20:07:42.411531 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 20:07:42.440248 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 686: this message will repeat every 5th retry.
W20240501 20:07:42.466176 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:42.481559 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:42.523197 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:42.563508 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:42.568665 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:42.669772 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:42.685156 28461 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:42.768349 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:42.797560 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:42.956677 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:42.989645 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:43.031955 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:43.070156 28362 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:43.080024 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:43.080837 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:43.115052 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:43.132047 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.222939 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 20:07:43.223668 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.249178 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.436592 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.461838 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:43.473549 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:43.494220 28362 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:43.498414 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:43.549654 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.571848 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:43.667654 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.696816 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:43.797858 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.865372 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:43.865460 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:43.896454 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.920235 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 20:07:43.920859 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.926498 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.932618 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.987496 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:43.996011 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:44.004923 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:44.011194 28500 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:44.032126 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:44.032636 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:44.036461 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 20:07:44.057442 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 20:07:44.127753 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 20:07:44.158639 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:44.256109 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:44.260327 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:44.332537 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:44.386488 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:44.422736 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 20:07:44.435056 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:44.437803 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:44.445427 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:44.496191 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:07:44.526118 28500 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:44.570078 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:44.642068 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:44.684877 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:07:44.725435 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:44.737255 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 20:07:44.910195 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:44.929143 28500 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:44.932288 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 20:07:44.937193 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:44.993714 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:45.069664 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:45.077281 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 691: this message will repeat every 5th retry.
W20240501 20:07:45.100931 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:45.105116 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:45.148645 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:45.218098 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:45.302976 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:45.352705 28511 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:45.369627 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:45.479169 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:45.488505 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:45.576020 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:45.578069 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:45.624207 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:45.726505 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:45.792197 28511 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:45.801103 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:45.857901 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:45.910732 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:45.946117 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:45.958354 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:45.971149 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:46.047257 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.136081 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.155874 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:46.249421 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 20:07:46.250499 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240501 20:07:46.302748 28562 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:46.333985 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.346009 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.348111 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:46.362360 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:46.391117 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.394402 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:46.398061 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.400547 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.407501 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.442801 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.568311 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.596176 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 20:07:46.619230 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.657373 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.693503 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:46.762233 28562 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:46.762862 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:46.771297 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:46.854331 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:46.920979 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:46.978323 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:47.040110 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
I20240501 20:07:47.188266 28562 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:47.196067 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:47.220059 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:47.229564 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:47.240243 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:47.244841 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:47.324920 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 20:07:47.374310 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:47.413717 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 696: this message will repeat every 5th retry.
W20240501 20:07:47.427316 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:47.440379 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:47.486229 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 20:07:47.605358 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:47.687105 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:47.724933 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:47.733237 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:47.754559 28511 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:47.781637 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:47.839053 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:48.031760 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:48.083681 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.135807 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.174129 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:48.184301 28562 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:48.189716 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:48.292109 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:48.295502 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 20:07:48.295563 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:48.316108 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.358223 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:48.451896 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.456316 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.523995 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.591306 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:48.592329 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:48.624804 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.637015 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240501 20:07:48.657763 28562 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:48.663251 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.673667 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.726868 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:48.754175 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:48.798341 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:48.873129 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.942335 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:48.998016 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 20:07:49.068730 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:07:49.069516 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:49.072281 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:49.079257 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 20:07:49.079787 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:49.079870 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:49.091329 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240501 20:07:49.223346 28562 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:49.223836 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:49.249486 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:49.252765 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:49.274489 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:49.410178 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:49.419191 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:49.426690 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:49.525741 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:49.694537 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:49.709723 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:49.728315 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 701: this message will repeat every 5th retry.
W20240501 20:07:49.732893 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 20:07:49.732878 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:07:49.767428 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240501 20:07:49.784106 28562 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:49.804817 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:49.831202 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:49.852952 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:49.892807 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:50.096568 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:50.119773 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:50.125726 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
W20240501 20:07:50.202576 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:50.229882 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:50.251870 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:50.273281 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:50.275282 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:50.327306 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:50.387888 28562 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:50.423629 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:50.434110 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:50.552153 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:50.638549 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:50.675798 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:50.725085 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:50.820871 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:50.838228 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:50.850181 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:50.965425 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 20:07:50.985677 28562 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:51.023114 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.030126 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:51.092537 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.109833 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.117597 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:51.164021 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:51.199548 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:51.211736 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:51.235881 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:51.360419 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:07:51.364524 28647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:51.401578 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.431114 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.433197 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 20:07:51.476963 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.488096 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.522648 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.528658 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:51.591857 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:51.662851 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.681421 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.706329 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:51.840065 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:51.841113 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:51.888166 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 20:07:51.950949 28647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:52.002732 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:52.017323 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:52.058214 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:52.080806 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:52.138649 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 20:07:52.152093 28669 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:52.152222 28669 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:52.152642 28669 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:52.153692 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:52.153842 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:52.153905 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:52.154052 28669 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:07:52.235890 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:52.298339 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:52.304553 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:52.346720 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:52.403576 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:52.520874 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:52.534654 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 20:07:52.536100 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:52.536885 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
W20240501 20:07:52.546337 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 706: this message will repeat every 5th retry.
I20240501 20:07:52.559284 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:52.584482 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:52.656484 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:52.783136 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:52.794791 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:52.886679 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:52.924055 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:52.973016 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:53.007902 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.040791 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:07:53.086565 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:53.149235 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:53.205483 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:53.231266 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:53.298998 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.299686 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:53.331992 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 20:07:53.347733 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.357609 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:07:53.501530 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.536904 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:07:53.571835 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:53.575268 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:53.592152 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:53.597905 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:53.622817 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.650147 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:53.654515 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:53.711620 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.740478 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.775534 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 20:07:53.816433 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:53.824785 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.863766 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.897539 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:53.952337 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.007257 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:54.032770 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.090667 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 20:07:54.091352 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240501 20:07:54.101694 28735 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:54.130430 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:54.143569 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:07:54.164541 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:54.195511 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:54.338927 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.370275 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 20:07:54.409700 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:54.421684 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.465795 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.473641 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.507162 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.578847 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:54.579815 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:54.658288 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:54.685511 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:54.757861 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.813155 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.817512 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:54.827172 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:07:54.854346 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 20:07:54.857849 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:54.966509 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:55.012830 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:55.042212 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 711: this message will repeat every 5th retry.
I20240501 20:07:55.103950 28761 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:07:55.104084 28761 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:07:55.104521 28761 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:07:55.105590 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:55.105914 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:55.105988 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:07:55.106132 28761 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:07:55.110441 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:55.173748 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:55.188823 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 20:07:55.279819 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:55.345503 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:55.444041 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:55.483191 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:55.491473 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:55.542423 28791 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:55.591204 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:55.657320 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 20:07:55.699977 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:55.794955 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:55.848731 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:55.895001 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:55.895203 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:55.898581 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:07:55.938149 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:55.954869 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:55.974885 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:56.000682 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.058245 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.069306 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:56.075928 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.094362 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.103880 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.153364 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:56.218699 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.231629 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 20:07:56.232519 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:56.234887 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:56.270814 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.357755 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.365592 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.475102 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:56.520985 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:07:56.540583 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:56.660413 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:56.677887 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.694098 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:56.719105 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.725790 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:56.768718 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 20:07:56.804383 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.820282 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.892030 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:56.998495 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:57.013053 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:57.141480 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:57.151186 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:57.151917 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:07:57.156586 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:57.284024 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:57.288111 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:57.402040 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:57.405601 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:57.457509 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 20:07:57.528388 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:07:57.599491 28846 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:57.605180 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:57.634910 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 20:07:57.711474 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:57.738044 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:57.755892 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 716: this message will repeat every 5th retry.
W20240501 20:07:57.865535 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:07:57.996882 28846 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:58.006219 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.017961 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.053618 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:58.128453 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:58.218346 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:58.225419 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:58.267588 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 20:07:58.299952 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.323496 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.333269 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:58.335534 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:58.353375 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.377072 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 20:07:58.381451 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.423187 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240501 20:07:58.424536 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:58.443037 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 20:07:58.449095 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.499423 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:58.514890 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.519292 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.530153 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.598270 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.603700 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.618521 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:58.766852 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:58.785997 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.786962 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:58.882326 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:58.913517 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:07:58.949702 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:58.951625 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:59.080358 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:59.120611 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:59.121268 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:59.122954 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:07:59.138088 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 20:07:59.227788 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240501 20:07:59.243155 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:59.344555 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:59.398113 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:59.409227 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:59.419453 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:59.481689 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240501 20:07:59.516229 28724 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:59.565816 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:59.626848 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:59.651011 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:59.659584 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:07:59.668475 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:07:59.711866 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:07:59.837630 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 124 similar messages]
I20240501 20:07:59.916810 28914 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:07:59.940822 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:00.017231 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:08:00.098855 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 20:08:00.108667 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:00.152133 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 20:08:00.154556 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:00.155630 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:00.304549 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 721: this message will repeat every 5th retry.
W20240501 20:08:00.363631 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:00.381086 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:00.396463 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:08:00.422466 28914 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:00.463747 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:00.652207 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:00.669055 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:00.724359 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:00.726405 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:00.752331 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:00.754007 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:00.758481 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:08:00.773203 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:00.795591 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:08:00.798625 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:00.923813 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:08:00.939541 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:00.958837 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:08:01.016377 28953 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:01.065891 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 20:08:01.125602 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:01.135799 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20240501 20:08:01.136510 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:08:01.139765 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 20:08:01.140286 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:01.168787 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.172601 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:01.211761 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.284409 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.301839 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:01.301822 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.351308 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.397805 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.513226 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:01.513226 28914 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:01.531872 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.611047 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:01.611749 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.638793 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.666086 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 20:08:01.679383 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:01.770728 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.777556 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:01.899233 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:01.985442 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:08:01.998389 28914 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:02.018287 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:02.022563 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:02.034183 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:02.159071 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:02.196017 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:02.282640 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:02.284314 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:02.294871 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:02.342734 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 20:08:02.448257 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:02.494633 28914 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:02.508399 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:02.512475 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 20:08:02.520258 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:08:02.633442 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 726: this message will repeat every 5th retry.
W20240501 20:08:02.638931 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:02.681651 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:02.849789 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:02.914448 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240501 20:08:02.920323 28953 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:02.937913 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:02.958566 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:03.000340 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.007604 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.051234 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:03.111828 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:03.139192 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:03.236248 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.244694 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.309381 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.333258 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.339252 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.355782 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:03.380836 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:08:03.482501 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:03.499830 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:08:03.501174 28953 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:03.534298 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:03.574065 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.628835 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:03.630070 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.669813 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 20:08:03.707820 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.715795 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:03.809533 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.834208 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 20:08:03.886139 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:03.901525 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:03.992019 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:04.013146 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 20:08:04.027155 29035 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:04.028191 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:04.060849 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:04.081878 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:04.098743 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:04.100562 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:04.188047 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 20:08:04.190971 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:04.228571 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 20:08:04.263978 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 20:08:04.264585 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:04.291235 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:04.362183 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:04.374447 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:04.399905 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:04.567651 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:08:04.581559 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:04.593338 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:04.628736 29035 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:04.721923 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:04.840237 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:08:04.893508 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:04.927855 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:05.028097 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:05.035166 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 20:08:05.049839 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
I20240501 20:08:05.057360 29053 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:05.117236 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:05.167109 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 731: this message will repeat every 5th retry.
W20240501 20:08:05.199709 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:05.297536 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:05.369637 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:05.426921 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:08:05.442066 29035 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:05.451129 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:05.488499 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:05.538156 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:05.550688 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:05.636831 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:05.646185 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:05.710640 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:05.719028 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:05.719396 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:05.736569 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:05.749641 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:05.926978 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:05.944013 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:08:05.991703 29035 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:06.079123 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 20:08:06.083117 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:06.097208 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:06.169461 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:06.172411 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.198252 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.208838 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.216568 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.290493 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.294493 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:06.325525 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:06.357162 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.373323 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 20:08:06.450498 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.480530 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:08:06.508961 29102 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:06.540520 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.635726 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.644052 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:06.654410 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.664532 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:06.691483 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 20:08:06.715150 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:06.718693 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.764837 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.775585 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.790062 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:06.881584 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:06.920675 29035 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:07.047221 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:07.118755 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:07.143877 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:07.256557 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:07.290539 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 20:08:07.320241 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:07.356657 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:07.376117 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 20:08:07.466111 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:07.511649 29129 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:07.644043 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 736: this message will repeat every 5th retry.
W20240501 20:08:07.674429 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:07.686388 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:07.753615 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:07.763566 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:07.834450 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:07.987778 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:08.006050 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:08.019127 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:08.108620 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:08:08.124394 29147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:08.177810 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.196090 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.203841 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.222399 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.227973 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:08.260465 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.286161 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:08.309386 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:08.405573 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 20:08:08.465461 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 20:08:08.470839 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.482806 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.493191 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:08.536546 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.561964 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 20:08:08.588866 29147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:08.694139 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:08.706754 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:08.709758 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.817327 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.817634 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:08.840342 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.859864 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.878352 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.879595 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.884829 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:08.896992 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:08.988410 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:08:09.011426 29169 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:09.021740 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:09.155102 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:09.171429 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 20:08:09.183244 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:09.188225 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:08:09.188805 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:09.225679 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:09.258262 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:09.272670 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:08:09.273386 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:09.298720 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:09.306677 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:09.351364 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 20:08:09.458369 29169 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:09.593688 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:09.635177 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:09.731124 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:09.804201 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:09.811059 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:09.820261 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 20:08:09.866196 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
W20240501 20:08:09.881377 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:09.922535 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:08:09.969936 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:08:09.974828 29169 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:09.975896 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 741: this message will repeat every 5th retry.
W20240501 20:08:10.134640 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:10.224601 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:10.247627 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:10.352914 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:08:10.367410 29147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:10.403288 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:10.480775 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:10.598202 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:10.621500 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:10.622587 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:10.647145 29209 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:08:10.647326 29209 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:08:10.647799 29209 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:08:10.648797 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:10.648974 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:10.649027 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:08:10.649168 29209 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:08:10.673228 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:10.719053 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:10.736018 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:10.809340 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:10.813571 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:10.814316 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:10.814517 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:10.825907 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 20:08:10.851384 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:10.861196 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:10.973942 29147 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:11.003427 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.013955 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 20:08:11.097730 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.122022 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:11.151749 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:11.154664 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:11.168651 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.362658 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:08:11.394574 29227 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:11.424638 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:11.466231 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.475937 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.483176 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.491626 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.514456 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.564606 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.592602 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 20:08:11.595722 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:11.627295 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:11.664695 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.668085 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.706245 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.752369 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.762567 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:11.810007 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:11.856115 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:11.880502 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240501 20:08:12.010267 29227 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:12.058010 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:12.078369 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:12.093196 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:12.168121 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:12.230505 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:12.285660 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:12.362310 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 20:08:12.390887 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 746: this message will repeat every 5th retry.
I20240501 20:08:12.439318 29227 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:12.490357 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:12.514202 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:12.629477 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 20:08:12.674129 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:12.760041 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:12.775224 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:12.791931 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:12.814214 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:12.861737 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
I20240501 20:08:12.938535 29254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:12.941407 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:12.960475 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.030459 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.059031 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:13.233373 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:13.241988 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:13.251796 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.296027 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:08:13.316598 29254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:13.324625 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.342406 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:13.362200 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.387583 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:13.419425 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:08:13.532816 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.556077 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 20:08:13.560786 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.655515 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:13.666712 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:08:13.706182 29227 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:13.738090 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.757668 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:13.767869 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.807641 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:13.853533 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.871737 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:13.904356 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:13.931375 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:14.012702 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:14.016288 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:14.073809 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:14.084491 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:14.105074 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:08:14.146260 29254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:14.157917 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:14.171257 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 20:08:14.273890 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:14.290503 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 20:08:14.291080 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:14.322381 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 20:08:14.339849 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 113 similar messages]
W20240501 20:08:14.342557 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:14.388435 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:14.430599 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:14.487380 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:08:14.543911 29254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:14.637779 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:14.666859 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:14.679847 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:14.742687 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:14.817900 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:14.861943 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:14.894447 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 20:08:14.944295 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:08:14.954246 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 751: this message will repeat every 5th retry.
W20240501 20:08:14.986311 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:15.035883 29227 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:15.103431 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:15.116890 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 20:08:15.267890 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:15.304960 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:15.325963 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:15.381083 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:15.392467 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:08:15.449913 29254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:15.504560 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:15.510249 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:15.512164 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:15.550053 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:15.706591 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:15.713191 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:15.732364 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:15.790810 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:08:15.840512 29227 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:15.876041 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:15.886754 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:15.977985 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:15.982141 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:15.983281 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:15.985471 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.005635 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.064575 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:16.095028 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 20:08:16.111052 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 20:08:16.239588 29238 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:08:16.239732 29238 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:08:16.240128 29238 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:08:16.240859 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:16.240963 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:16.241062 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:08:16.241189 29238 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:08:16.291641 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:16.361114 29254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:16.374491 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.383275 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:16.417054 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.436755 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 20:08:16.448827 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:16.454139 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.466270 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.482597 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.489876 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.494266 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:16.536465 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.543226 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.579463 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.661912 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.865257 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:16.869131 29254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:16.911924 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:16.960844 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:16.966184 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 20:08:16.968159 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:17.024761 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:17.088088 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:17.243722 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 20:08:17.251142 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:17.259876 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:08:17.286048 29254 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:17.329335 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:17.344229 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:17.556617 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 20:08:17.609133 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:17.628332 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:17.711910 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:17.716673 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:17.720402 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 756: this message will repeat every 5th retry.
W20240501 20:08:17.725082 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:17.753783 29384 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:17.910828 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:17.957625 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:17.967500 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:18.012635 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:18.013914 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.050113 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.110168 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.117929 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:18.211448 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.251274 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:08:18.273841 29421 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:18.292709 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:18.319028 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:18.376519 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:18.433398 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 20:08:18.493214 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.498538 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:18.501657 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.533823 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.544278 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 20:08:18.544514 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.608917 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:08:18.734786 29421 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:18.742262 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:18.758859 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 20:08:18.773710 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.779067 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:18.880494 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:18.905567 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.918841 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:18.945777 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.048082 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.089979 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.114723 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:19.119669 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:19.132784 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.226641 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.237655 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
I20240501 20:08:19.239923 29421 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:19.261961 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.262557 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.265973 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:19.339993 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 20:08:19.340768 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:19.486485 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 20:08:19.487241 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.540164 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:19.619024 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:19.627894 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 20:08:19.665684 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.751745 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:19.793524 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
I20240501 20:08:19.830989 29467 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:19.840348 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:19.887688 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:19.960222 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 20:08:19.989023 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:08:20.061269 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:20.174552 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:20.210006 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:20.219301 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:20.258157 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:20.302878 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 761: this message will repeat every 5th retry.
W20240501 20:08:20.310245 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:20.325122 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:08:20.402750 29467 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:20.472944 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:20.484434 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:20.507278 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:20.512495 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:20.550587 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:20.612205 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:20.623483 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:20.669137 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:20.715757 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:20.752765 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:20.799291 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:20.902012 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 20:08:20.918992 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:20.971063 29467 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:21.067042 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.087385 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:21.106786 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.118173 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.145823 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:21.155870 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:21.224330 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.277341 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 20:08:21.367112 29498 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:21.373754 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:21.415012 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.416388 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.416986 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.429387 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 20:08:21.471758 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.598984 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.628024 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.631489 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:21.634871 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.640990 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 20:08:21.649888 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:21.672319 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.683089 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.710345 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:21.872488 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:21.885097 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:21.925721 29526 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:21.928515 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
I20240501 20:08:21.961475 29238 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:08:21.961613 29238 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:08:21.962087 29238 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:08:21.963042 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:21.963232 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:21.963313 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:08:21.963446 29528 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:08:22.140849 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:22.237943 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:22.292385 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:22.322360 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:22.337365 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240501 20:08:22.374212 29526 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:22.389233 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:22.426674 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:22.546404 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 20:08:22.634841 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:22.657267 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:22.676357 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 766: this message will repeat every 5th retry.
W20240501 20:08:22.692504 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:22.752390 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:22.793474 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:22.795550 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:22.805728 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:22.821264 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:08:22.936573 29498 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:22.997522 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:23.012708 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:23.082168 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.237499 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:23.252696 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.279997 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:23.303191 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:23.342090 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:23.388062 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:23.391528 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:08:23.495862 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:23.507920 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.571702 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.575168 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 20:08:23.575812 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.631969 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.632572 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:23.671662 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 20:08:23.703017 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.721140 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.726362 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:23.736524 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.778784 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.822479 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.845232 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.847066 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:23.862027 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:23.874397 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:23.935061 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:23.961386 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:23.964838 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:24.014194 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:24.085996 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:24.116333 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:24.168000 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:24.201822 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 20:08:24.210124 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:24.235653 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:24.395751 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:24.411526 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 20:08:24.412153 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:24.440649 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
I20240501 20:08:24.465324 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:24.505383 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240501 20:08:24.583175 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:24.607789 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:24.633091 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:24.659389 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 20:08:24.707935 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:24.901842 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:24.939474 29599 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:24.972183 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:25.053833 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:08:25.118618 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 771: this message will repeat every 5th retry.
W20240501 20:08:25.126188 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:25.227929 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:25.244526 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:25.318773 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:25.323933 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 20:08:25.327980 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:25.357360 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:25.389442 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 20:08:25.411521 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:25.414820 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:25.433187 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:25.490712 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:25.506484 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:25.647758 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:25.707057 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:25.758188 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:25.798539 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:25.854069 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:25.879438 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:25.896386 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:25.955090 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.030867 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.054356 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:26.079768 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:26.086791 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.095822 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.146123 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 20:08:26.172652 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.179356 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.184940 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:26.259830 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:26.265890 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.283581 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 20:08:26.332907 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:26.394258 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.402154 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:26.470676 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:26.470911 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.502537 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.558666 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.585704 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:26.618575 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 20:08:26.681718 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:26.689654 29563 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:08:26.689791 29563 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:08:26.690192 29563 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:08:26.691288 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:26.691401 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:26.691462 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:08:26.691718 29563 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:08:26.763119 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:26.800177 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:26.829304 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.847806 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.876489 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.881354 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:26.883150 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:26.975244 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:27.125134 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:27.148313 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:27.198894 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 20:08:27.309628 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 20:08:27.329854 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:27.341426 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:27.374523 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:27.379845 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:27.401024 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:27.405272 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:27.564731 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 776: this message will repeat every 5th retry.
W20240501 20:08:27.641192 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:27.642324 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:27.700021 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 20:08:27.807829 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:27.812525 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:27.832747 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:27.850226 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:27.853104 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:27.869331 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:08:27.904371 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:28.143991 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:28.175408 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:28.317966 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:28.335811 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.335846 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.368075 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:28.377637 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:28.382845 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.396561 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:28.413826 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.427471 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.427937 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.447562 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:28.544260 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:28.667013 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 20:08:28.667642 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 69 similar messages]
I20240501 20:08:28.752666 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:28.765317 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:28.797901 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.803158 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.819036 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.861209 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.905624 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:28.963157 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:28.976357 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:28.978436 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:29.040308 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:29.058630 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:29.113236 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:29.139771 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:29.259146 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:29.260342 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 20:08:29.277805 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:29.348251 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:29.351565 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 20:08:29.351996 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:29.412441 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:08:29.437638 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:29.438091 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:29.459558 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:29.517613 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 20:08:29.558750 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:29.574232 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:29.700155 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:29.735209 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:29.742318 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 20:08:29.823693 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:29.878778 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:29.885179 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 20:08:29.925540 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:30.024667 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:30.060106 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 119 similar messages]
W20240501 20:08:30.124828 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:30.125913 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:30.149435 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 781: this message will repeat every 5th retry.
W20240501 20:08:30.163275 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:30.176434 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:30.183137 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 20:08:30.220029 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:30.269526 29575 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:30.341131 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:30.357353 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:30.371714 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:30.496505 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:30.619937 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:30.691571 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:30.764101 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:30.774135 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:30.804986 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:30.819133 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:30.824568 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:30.856874 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:08:30.888839 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:30.952512 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:30.961795 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.086617 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.090831 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:31.096020 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.173663 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:31.182528 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.269618 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:31.326450 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 20:08:31.369861 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:31.376341 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:31.428053 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.438669 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:31.498579 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.525604 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.536885 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.546239 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.582648 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.672847 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:31.675132 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.708825 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.720072 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
I20240501 20:08:31.756564 29767 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:31.760370 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:31.766531 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 20:08:31.909803 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:31.913250 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:32.006361 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:32.051620 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:32.128127 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:32.146109 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:32.177592 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:32.212952 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:08:32.224474 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:32.278641 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:32.355970 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 20:08:32.376936 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:32.467666 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:32.527750 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 20:08:32.559477 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:32.592677 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 20:08:32.600761 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:32.624846 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:32.732679 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:32.734408 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 786: this message will repeat every 5th retry.
W20240501 20:08:32.773910 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:32.820791 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:32.824807 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:32.925278 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:32.960661 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:33.006198 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:33.057489 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:08:33.118629 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:33.169885 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.212641 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:33.269518 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.341204 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:33.352397 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.401347 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:33.416239 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.416808 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:33.452731 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.484757 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:33.521301 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.607805 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:33.609601 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:08:33.658362 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:33.694964 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:08:33.733637 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.740489 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:33.862509 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 20:08:33.878700 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:33.913967 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.936444 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.952036 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.964421 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:33.996035 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:34.017185 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 20:08:34.071748 29814 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:34.082962 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:34.111969 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:34.143929 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:34.144630 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:34.174216 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:34.275166 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 20:08:34.292572 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:34.449542 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 20:08:34.450125 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:34.452461 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:34.489573 29647 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:34.565279 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 20:08:34.703507 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:34.729737 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:34.734844 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 20:08:34.768443 29563 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:08:34.768608 29563 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:08:34.769021 29563 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:08:34.769920 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:34.770119 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:34.770198 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:08:34.770332 29563 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:08:34.775732 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:34.811306 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 20:08:34.811547 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:34.839568 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:34.841097 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:35.012511 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 791: this message will repeat every 5th retry.
W20240501 20:08:35.019467 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:35.051527 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:35.058633 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:08:35.066267 29878 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:35.073534 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 20:08:35.081148 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:08:35.081708 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:35.120894 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 20:08:35.127630 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:35.242555 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:35.291846 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:35.346586 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:35.447604 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:35.478565 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:35.479146 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:35.489678 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:35.505312 29879 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:35.737123 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:35.760191 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:35.789647 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:35.840068 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:35.875850 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:35.891515 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:08:35.916287 29878 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:35.958884 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:35.967298 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:35.990583 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:36.013912 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:36.047502 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.097772 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.170081 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.290191 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.300180 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.331763 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:36.343879 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 20:08:36.415163 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.425886 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:36.432694 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:36.441514 29878 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:36.450176 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.498554 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.565960 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:36.604382 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.614406 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.653016 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.657863 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 20:08:36.762182 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.817739 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:36.833596 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:36.847719 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:36.862152 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:36.918175 29878 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:37.015537 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:37.110555 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:37.152920 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:37.251304 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:37.257756 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:37.259874 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:08:37.303782 29924 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:37.366693 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:37.398195 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:37.413722 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:37.428929 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:37.432557 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 20:08:37.504609 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 20:08:37.522811 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 20:08:37.527799 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 796: this message will repeat every 5th retry.
W20240501 20:08:37.625898 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:37.632959 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:37.770655 29878 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:37.918061 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:37.941790 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:37.963750 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:37.969745 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:37.993168 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:38.005656 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:38.105196 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:38.110684 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:38.183615 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:38.192245 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:08:38.258708 29924 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:38.274132 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:38.415534 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:38.455204 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:38.483598 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:38.484568 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:38.531450 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:38.540664 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:38.597440 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:38.615216 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:38.619302 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 20:08:38.630715 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:38.701942 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:38.703467 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 20:08:38.718194 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:08:38.733916 29878 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:38.788628 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:38.880429 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:38.898588 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:38.922572 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:39.016043 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:39.028445 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:39.055189 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:39.061717 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:39.069173 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:39.099897 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:39.135654 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:39.174322 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 20:08:39.319938 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:39.343997 29989 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:39.384054 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:39.402781 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:39.462167 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:39.509706 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 20:08:39.512792 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:39.588027 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:08:39.611991 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:39.688558 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:39.757858 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:39.758631 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:39.767767 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:39.783401 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 20:08:39.797874 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 20:08:39.806087 29878 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:39.808176 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 801: this message will repeat every 5th retry.
W20240501 20:08:39.880175 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:39.889935 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:39.972491 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 20:08:40.029426 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:40.036485 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:40.172593 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:08:40.231967 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:40.255168 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
I20240501 20:08:40.302126 29989 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:40.310078 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:40.348305 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:40.379698 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:40.388197 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:40.449615 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:40.656392 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:40.683552 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:40.688482 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:40.762763 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240501 20:08:40.782552 29878 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:40.810230 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:40.830924 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:40.870998 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:40.914431 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:40.916209 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:40.932505 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.002811 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:41.038993 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.166980 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 20:08:41.259183 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.263016 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.275321 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.284853 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.289523 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.311332 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.312474 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:08:41.349591 30045 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:41.408989 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:41.446377 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:41.458386 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:41.473914 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.552196 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.566578 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.586642 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:08:41.620117 30033 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:08:41.620263 30033 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:08:41.620735 30033 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:08:41.621975 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:41.622072 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:41.622189 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:08:41.622335 30033 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:08:41.623184 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:41.652784 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.726300 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 20:08:41.764727 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:08:41.908454 30045 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:41.952379 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:41.963378 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:41.977299 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:41.985411 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:42.128396 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:42.148301 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:42.149606 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 806: this message will repeat every 5th retry.
W20240501 20:08:42.162072 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:42.312179 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 20:08:42.324107 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 20:08:42.342484 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:42.351517 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:42.397837 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:08:42.428258 30061 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:42.527021 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:42.590988 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:42.689278 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:42.731961 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:42.769207 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:42.862116 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:42.936281 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:08:42.985780 30088 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:43.024114 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:43.041565 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.065682 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 20:08:43.125203 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:43.143662 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.157579 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:43.231986 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:43.258636 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:43.326213 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:43.388602 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.394003 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.411051 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 20:08:43.422856 30088 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:43.503353 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.535177 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.580909 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:43.584519 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:43.656430 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.671770 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.672318 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:43.693519 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.699050 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.734377 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.767716 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:08:43.773244 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.867519 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.869954 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.876770 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:43.898792 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
I20240501 20:08:43.911417 30061 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:43.938971 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:43.984006 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:44.059975 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:44.062882 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:44.110680 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:44.217216 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:44.242609 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 20:08:44.288177 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:44.359072 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:44.414204 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:08:44.485958 30128 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:44.553926 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 115 similar messages]
W20240501 20:08:44.569406 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:44.604935 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:08:44.613462 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 811: this message will repeat every 5th retry.
W20240501 20:08:44.619362 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:44.636044 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:44.689090 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:44.768808 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 20:08:44.771025 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 20:08:44.847894 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:08:44.874032 30061 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:44.894985 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:44.929836 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:45.156441 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:45.247709 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 20:08:45.248381 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:45.258090 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:45.275256 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:45.283777 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:08:45.343010 30128 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:45.353458 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:45.404224 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:45.422041 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:45.482589 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 20:08:45.591449 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:45.604449 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:45.723027 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:45.733603 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:45.761876 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:45.766126 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:45.790136 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 20:08:45.791735 30061 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:45.825839 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:45.851007 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:45.860524 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:45.866608 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:45.891165 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:45.934873 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:45.990914 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:46.026502 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.117013 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:46.117435 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.138423 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.190964 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:08:46.239392 30061 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:46.293700 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.298776 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 20:08:46.334523 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.335876 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.341567 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.354382 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.423381 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:46.436554 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:46.471452 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.517092 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:46.569118 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:46.571532 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:46.644508 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:08:46.737423 30061 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:46.816485 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 20:08:46.852173 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:46.960078 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:46.987416 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:47.008381 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:47.071163 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:08:47.176837 30061 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:47.180294 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:47.182696 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:47.220494 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 20:08:47.221969 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 816: this message will repeat every 5th retry.
W20240501 20:08:47.333885 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 20:08:47.384001 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:47.388890 30033 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:08:47.389014 30033 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:08:47.389427 30033 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:08:47.390347 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:47.390455 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:47.390533 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:08:47.390677 30033 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:08:47.438732 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:47.511596 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:47.607594 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:47.618433 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:47.635928 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:08:47.673010 30128 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:47.698002 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:47.726008 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:47.803422 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:47.819016 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:48.001416 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:48.021485 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.087170 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
I20240501 20:08:48.218587 30224 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:48.221443 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:48.223306 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:48.230548 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.268306 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:48.282559 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.311772 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:48.326679 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:48.345186 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:48.456984 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.462121 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.527956 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:48.566642 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.575922 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.597795 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.612589 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.615382 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:48.643065 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:08:48.674608 30224 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:48.691066 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 20:08:48.716789 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.788614 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.811547 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:48.834146 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.889533 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:48.954422 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.963093 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 20:08:48.963660 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:48.994836 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:49.014935 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:49.045083 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 20:08:49.080267 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 20:08:49.086153 30224 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:49.130642 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:49.238283 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:49.294992 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:49.354281 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:49.401548 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:49.486485 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:49.523885 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 821: this message will repeat every 5th retry.
W20240501 20:08:49.544226 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:49.556782 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 20:08:49.566859 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 20:08:49.687525 30061 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:49.699717 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:08:49.700277 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:49.731004 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 20:08:49.745379 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 20:08:49.835096 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:49.864238 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:49.918272 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:50.019923 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:50.091367 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:50.160583 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:50.185971 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:50.188517 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:50.262952 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 20:08:50.263613 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:50.264052 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:08:50.297168 30261 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:50.301002 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:50.353124 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:50.548046 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:50.643023 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:50.644704 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:50.663517 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:50.692768 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:50.712635 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:50.735101 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:50.820129 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 20:08:50.827430 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:50.830485 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:50.849241 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:08:50.866887 30261 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:50.968956 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:50.993913 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.100932 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.127105 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:51.129938 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.150552 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.169978 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.223665 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:51.242439 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 20:08:51.242566 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:51.285784 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.291249 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 20:08:51.337477 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.357307 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:51.370000 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.383263 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:51.393759 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:08:51.401407 30261 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:51.444129 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.531929 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.593231 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.670701 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.720073 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:51.739009 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 20:08:51.829936 30261 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:51.838963 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:51.839926 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:51.841841 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:51.881618 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:51.899811 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:52.087764 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 826: this message will repeat every 5th retry.
W20240501 20:08:52.145798 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 20:08:52.224131 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:52.300863 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:52.307358 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
I20240501 20:08:52.314157 30334 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:52.326459 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:52.355355 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:52.377722 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:52.536824 30033 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:08:52.536967 30033 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:08:52.537387 30033 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:08:52.538478 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:52.538599 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:52.538661 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:08:52.538817 30033 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:08:52.619102 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:52.706820 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:52.751945 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:52.768676 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:08:52.770917 30261 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:52.783819 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:52.858302 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:52.918421 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:53.007054 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.014894 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.065203 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:53.097126 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.127331 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:53.179566 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:08:53.198413 30261 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:53.250032 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:53.263156 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:53.268711 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.358537 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.378960 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.387918 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:53.469751 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:53.518009 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.528151 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:53.545914 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 20:08:53.610776 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.621905 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.639942 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.657975 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:53.663690 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 20:08:53.687309 30261 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:53.735471 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:53.740438 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:53.778530 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.839193 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:53.855952 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 20:08:53.885543 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 20:08:53.969338 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:08:53.982946 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:54.015442 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:54.018852 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:54.122756 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:54.127187 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:54.168017 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:54.179116 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:54.192931 30334 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:54.202186 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:54.223387 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:54.267000 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:54.325271 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:54.354341 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:54.379390 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 20:08:54.517817 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:54.606920 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:54.619228 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:54.636603 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 831: this message will repeat every 5th retry.
W20240501 20:08:54.639496 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:08:54.676604 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:54.708724 30261 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:54.723758 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:08:54.743224 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:54.752674 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 20:08:55.058050 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:55.066926 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:55.094048 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:55.123258 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:55.141701 30261 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:55.200055 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:55.302328 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:55.363493 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:08:55.406293 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:55.440398 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:08:55.562297 12442 maintenance_manager.cc:419] P be1305e487d64b748456a721443e3b3b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033394
I20240501 20:08:55.565151 12441 maintenance_manager.cc:643] P be1305e487d64b748456a721443e3b3b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.003s user 0.002s sys 0.000s Metrics: {"bytes_written":1677,"delete_count":0,"lbm_write_time_us":110,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":75,"threads_started":1,"update_count":18,"wal-append.queue_time_us":179}
I20240501 20:08:55.565500 12442 maintenance_manager.cc:419] P be1305e487d64b748456a721443e3b3b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.116136
I20240501 20:08:55.569664 12441 maintenance_manager.cc:643] P be1305e487d64b748456a721443e3b3b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.004s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":14252,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":160,"lbm_read_time_us":164,"lbm_reads_lt_1ms":17,"lbm_write_time_us":166,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":112,"threads_started":2,"update_count":18}
W20240501 20:08:55.603330 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:55.624588 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:55.642066 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:55.664014 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:55.689659 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:55.694197 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:08:55.729233 30426 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:55.796062 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:55.797225 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:55.832927 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:55.836078 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:55.932685 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:55.982300 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.040199 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.074249 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.113574 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:56.149663 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 20:08:56.199100 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:56.221844 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.244683 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:56.259116 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.315232 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.318843 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.320909 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.325562 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:56.329272 30443 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:56.359856 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.402441 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.459523 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:56.547466 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 20:08:56.605847 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:56.679132 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.684917 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.703163 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 20:08:56.741544 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:56.801450 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:56.814364 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:56.814683 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.820356 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:08:56.823570 30443 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:56.850481 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:56.922319 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:56.941395 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 20:08:57.049784 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:57.183073 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 836: this message will repeat every 5th retry.
W20240501 20:08:57.203402 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:57.250238 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:57.258935 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 20:08:57.329625 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:57.398047 30476 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:57.401124 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:57.561371 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:57.579713 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:57.627841 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:57.691162 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:57.715936 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:08:57.796875 30443 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:57.888864 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:57.949208 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:58.000276 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.127813 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.135125 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.157367 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:58.192771 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:58.193132 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:58.204749 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:58.222499 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.251233 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.319073 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:08:58.381229 30498 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:58.381973 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:58.510174 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 20:08:58.573355 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.592444 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:58.606243 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.610736 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.623776 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:58.698632 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:58.711964 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.816859 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:58.834419 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:08:58.850437 30498 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:58.861392 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:58.870385 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.879041 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.883441 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:58.953915 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:59.009562 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:59.016747 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:59.041841 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:59.044654 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 20:08:59.055310 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 20:08:59.099292 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:59.137960 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:59.197697 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:08:59.253371 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:59.299644 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:08:59.309962 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:59.328243 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 20:08:59.341840 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 20:08:59.364110 30443 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:08:59.422494 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 20:08:59.535548 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:59.568179 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:59.576514 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:59.590183 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 841: this message will repeat every 5th retry.
W20240501 20:08:59.664407 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:59.679652 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 110 similar messages]
W20240501 20:08:59.702250 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:08:59.757480 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 20:08:59.830830 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:08:59.846377 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 20:08:59.886906 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:08:59.986089 30498 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:00.085379 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:00.096518 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:00.124419 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:00.158597 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:09:00.271167 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:00.302358 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:00.388180 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:09:00.430656 30443 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:00.449087 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:09:00.589923 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:00.593024 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:00.680572 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:00.708563 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:00.725963 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:09:00.738330 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:00.833770 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:00.838996 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:00.862156 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:00.862583 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:00.924716 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:00.932010 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:09:00.968371 30498 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:01.039229 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 20:09:01.122278 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:01.147435 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.158263 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:01.175191 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:01.182004 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 20:09:01.182668 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:09:01.235266 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:09:01.235728 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.256574 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.257851 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.320343 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:01.322573 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:09:01.328982 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.389168 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:09:01.403749 30573 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:01.446779 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.458590 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.510382 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:01.599020 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.648640 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:01.657709 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.665441 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:01.668465 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.750756 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:01.803186 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:01.818964 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 20:09:01.834553 30573 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:01.876263 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 20:09:01.948897 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 20:09:01.984124 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:01.993127 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 846: this message will repeat every 5th retry.
W20240501 20:09:02.026971 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:02.040535 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:02.174649 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 20:09:02.188360 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:02.201530 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:02.250258 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:02.282038 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 20:09:02.286048 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:02.375221 30573 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:02.561239 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:09:02.588385 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:02.589596 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:02.699129 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:02.729617 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:02.786579 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:09:02.793102 30573 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:02.818709 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:02.873353 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:02.901582 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:02.919397 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.052294 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:03.101073 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:03.134501 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:03.228089 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:03.239245 30628 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:03.316954 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.378959 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.392416 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.409513 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:03.429030 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.449416 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:03.472349 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.516428 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:03.554255 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.560117 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:03.657542 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.662772 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 20:09:03.668465 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:03.677565 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 20:09:03.753861 30573 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:03.755748 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:03.775902 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:03.847734 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.934862 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.936380 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.986727 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:03.991050 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:04.011399 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:04.061811 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:04.122956 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:04.145031 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:04.148236 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 20:09:04.150818 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:04.156100 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:04.206539 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:09:04.248925 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:09:04.306157 30666 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:04.317703 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 851: this message will repeat every 5th retry.
W20240501 20:09:04.332886 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:04.433640 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:04.497139 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:04.501015 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 20:09:04.506016 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:04.527395 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 20:09:04.557968 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:04.574551 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:09:04.654035 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:04.677726 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 20:09:04.705186 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 20:09:04.710335 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:04.727628 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:04.789856 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 20:09:04.808552 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 20:09:04.823392 30685 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:05.011185 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:05.099177 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:05.154937 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:05.175264 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:09:05.228724 30666 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:05.263015 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:05.309237 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:05.320606 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:05.347036 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:05.355391 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:05.531564 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 121 similar messages]
W20240501 20:09:05.557722 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:05.613173 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:05.679031 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:05.689677 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:05.691042 30666 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:05.723409 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:05.727936 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:05.753024 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:05.800976 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:05.849357 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.042507 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.085775 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:09:06.100813 30666 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:06.128333 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:06.129602 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:06.140201 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:06.141786 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:06.168938 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 20:09:06.184597 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:06.219722 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.241336 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:09:06.268172 30610 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:09:06.268309 30610 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:09:06.268713 30610 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:09:06.269891 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:06.270213 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:06.270300 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:09:06.270430 30725 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:09:06.275854 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:06.400097 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.484907 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.492045 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:06.511485 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 20:09:06.573225 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:06.604579 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:06.621927 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.660852 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.684652 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:06.700935 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:09:06.715418 30666 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:06.732671 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.736627 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.774616 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.791586 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:06.795006 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:06.835815 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 20:09:06.842188 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:06.955675 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 856: this message will repeat every 5th retry.
W20240501 20:09:07.002841 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:07.068358 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 20:09:07.154786 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:07.156184 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 20:09:07.219345 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:07.228067 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:07.231931 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:07.255101 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:07.327839 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:07.434280 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:07.612012 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:07.767581 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:07.796377 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:09:07.810823 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:07.814054 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:07.818014 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:07.850337 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:07.890072 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:07.924321 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.013823 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.030424 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.046975 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:08.100785 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.216796 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:08.233669 30769 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:08.243515 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:08.266148 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.295637 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:08.347399 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.460451 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:08.483034 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:08.560387 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.568783 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.581073 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 20:09:08.646214 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:08.658953 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:08.724025 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:08.732753 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:09:08.761373 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:08.784153 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.816684 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:08.935427 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:08.938791 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 20:09:08.991622 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.047576 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:09.079560 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:09.145923 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.145969 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:09.152099 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:09.175029 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
I20240501 20:09:09.175807 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:09.193563 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.205890 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:09.208150 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.239928 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 20:09:09.240674 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 20:09:09.263237 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.294472 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.350615 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 861: this message will repeat every 5th retry.
W20240501 20:09:09.401373 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.433655 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.465346 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:09.558645 30769 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:09.584998 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.658828 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:09.699422 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:09.709617 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 20:09:09.722440 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:09.779917 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:09.855525 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 20:09:09.885247 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 20:09:09.889245 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:09.896816 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240501 20:09:10.002487 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:10.128509 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:10.155503 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:10.188769 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:10.221372 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:10.254735 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:10.319563 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:10.354673 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:10.368521 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:10.441865 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:10.492251 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:10.522534 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:10.550415 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:09:10.551024 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:09:10.564479 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:10.583423 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:10.723955 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:10.746181 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:10.832854 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:10.900343 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
W20240501 20:09:10.946548 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:10.997721 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.000394 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:09:11.055199 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:11.059119 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:11.071606 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.087914 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.106254 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.122027 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 20:09:11.144099 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.165973 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:11.231060 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:11.311931 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:11.359824 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:11.492233 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.515408 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:11.529397 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:11.575395 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.608980 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:11.615995 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
I20240501 20:09:11.653493 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:11.663163 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:11.669785 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.701331 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 866: this message will repeat every 5th retry.
I20240501 20:09:11.743300 30807 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:09:11.743428 30807 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:09:11.743801 30807 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:09:11.744730 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:11.745430 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:11.745507 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:09:11.745697 30807 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:09:11.751211 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:11.767453 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.774796 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:11.788198 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:11.903086 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 20:09:12.045238 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:12.057466 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:12.059345 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:12.109088 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:12.122898 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 20:09:12.148047 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:12.155915 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:12.245254 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:12.249653 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:12.294656 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:12.406198 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 20:09:12.453709 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:12.522601 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:12.544158 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 20:09:12.552196 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:12.666771 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:12.676278 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:12.698545 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:12.706828 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:12.836268 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:12.872695 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:12.879691 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:12.963109 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:13.023416 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:13.070011 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:13.071847 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:13.074369 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:13.315014 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:13.329805 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:13.406016 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:13.443542 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 20:09:13.453440 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:13.457739 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:13.468605 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:13.469172 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:13.477699 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:13.561494 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:13.644599 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:13.671108 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:13.698537 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:13.738550 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:13.754166 30753 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:13.804942 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:13.822770 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 871: this message will repeat every 5th retry.
W20240501 20:09:13.916939 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:13.923338 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:13.941104 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 20:09:14.007391 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:14.030607 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:14.031199 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:14.098937 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:14.166064 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:14.216501 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:14.240878 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 20:09:14.258389 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:14.287561 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:14.302095 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:14.310007 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:14.320243 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:14.324993 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:14.357528 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:14.421284 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:14.458848 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:14.478399 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:14.537328 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:09:14.666146 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:14.727063 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 20:09:14.823616 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:14.824189 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:09:14.824844 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:14.830346 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:14.868669 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:14.912194 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 20:09:14.918314 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240501 20:09:14.922232 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:15.064002 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:15.108175 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:15.131700 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:15.180176 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 20:09:15.185753 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 20:09:15.195740 30967 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:15.209232 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:15.294975 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:15.336292 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:15.387018 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:15.429569 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:15.441064 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:09:15.601231 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:15.622141 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:15.631628 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:09:15.705654 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:15.716198 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:15.787580 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 20:09:15.860002 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:15.870028 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:15.882107 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:15.919176 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:15.958556 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:16.032312 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.040174 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:09:16.042150 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:16.112890 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:16.175328 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.207202 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:16.209355 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 876: this message will repeat every 5th retry.
W20240501 20:09:16.262323 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.297459 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:16.317819 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 20:09:16.329156 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:16.347844 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:16.361182 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:16.437223 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:16.500166 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.539232 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.611442 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.665205 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.672288 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:16.713702 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.714570 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:16.748096 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:16.793447 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:16.800235 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.821062 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:16.837203 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:16.858153 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:16.889317 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:09:16.914350 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:16.930773 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:17.189090 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:17.237100 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:17.274544 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:17.299590 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:17.301465 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:17.368804 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:17.405192 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:17.452353 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 20:09:17.477015 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:17.480170 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 20:09:17.526116 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:17.565264 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 20:09:17.774849 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:17.802224 30967 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:17.835671 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:17.837182 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:17.898116 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:17.928249 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:17.949743 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:17.987934 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.011082 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.088925 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.247608 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:18.292507 30967 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:18.306473 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:18.329166 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 20:09:18.340430 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:18.402994 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.456565 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:18.465785 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.547657 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.587788 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.650755 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:18.666501 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
I20240501 20:09:18.676467 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:18.677796 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:18.699896 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:18.727494 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:18.758390 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.770470 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.790993 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 20:09:18.796500 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.833705 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 881: this message will repeat every 5th retry.
W20240501 20:09:18.921671 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:18.934103 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:18.972383 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:18.974117 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:19.061964 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:19.088496 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:19.090451 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:19.155560 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:19.158403 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:19.184851 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:19.262319 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240501 20:09:19.316962 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:19.394222 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:19.419081 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:19.457096 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:19.492628 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:19.516194 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:19.526979 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:19.569792 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:19.665073 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:19.768357 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:19.790792 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 20:09:19.825665 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:19.845161 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 20:09:19.845731 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:19.962936 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 20:09:19.966197 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:19.966799 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:20.063875 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:20.072995 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:20.093499 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:20.183555 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 20:09:20.194638 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:20.226194 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 20:09:20.298702 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:20.316213 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:20.321221 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:09:20.349854 30901 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:20.427444 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:20.549065 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:20.584756 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:20.593230 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:20.643155 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:09:20.643849 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:20.651728 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:20.731299 31101 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:20.732951 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:20.793491 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:20.855160 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:20.888379 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 20:09:20.960537 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:20.983029 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.048902 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:21.244138 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 886: this message will repeat every 5th retry.
W20240501 20:09:21.251653 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:21.259083 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:21.283740 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:21.300444 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
I20240501 20:09:21.338249 31101 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:21.344069 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.365653 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.385530 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:21.443761 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.480537 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.490550 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:21.498539 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.509025 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.509162 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:21.533839 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.630837 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.635989 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 20:09:21.721518 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.743618 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:21.791836 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:21.828454 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:09:21.853701 31101 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:21.892324 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.919816 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:21.937611 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:21.957881 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:21.971104 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:22.088054 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:22.211522 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:22.217612 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:22.240650 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:22.317962 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:09:22.337006 31101 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:22.354197 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:22.442564 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:22.456754 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 20:09:22.487888 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:22.518767 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:22.581581 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:22.622130 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 20:09:22.631067 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:22.723321 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:22.739128 31140 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:22.762637 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:22.767897 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 20:09:22.862390 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:22.869794 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:22.915982 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:23.007612 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 20:09:23.113601 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:23.131934 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:23.201352 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:23.203263 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:23.209352 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:23.248889 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:09:23.304383 31177 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:23.315521 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:23.422243 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:23.480118 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:23.560341 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:23.685401 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 891: this message will repeat every 5th retry.
W20240501 20:09:23.717272 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:23.735554 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:23.784976 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:23.808506 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:23.834947 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:09:23.913147 31177 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:23.921133 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:23.928033 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:23.969700 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:23.979656 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:24.049482 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:24.057435 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:24.093586 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:24.104969 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 20:09:24.165948 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:09:24.172456 31020 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:09:24.172581 31020 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:09:24.172927 31020 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:09:24.173988 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:24.174089 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:24.174156 12921 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:09:24.174309 31020 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:09:24.182934 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:24.217456 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:24.321067 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:24.328682 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 20:09:24.343269 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:24.372475 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:24.379748 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:24.429610 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:24.502833 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:24.529197 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
I20240501 20:09:24.537883 31208 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:24.600200 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:24.674408 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:24.694975 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:24.790369 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:24.798204 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:24.806305 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:24.881008 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 20:09:24.888247 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:24.893358 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:24.923662 31195 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:24.993952 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:25.002707 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 20:09:25.003293 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:25.022657 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:25.085453 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 20:09:25.088629 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 20:09:25.218423 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:25.275816 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:25.292354 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:25.303668 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:25.313774 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:25.353143 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
I20240501 20:09:25.367781 31208 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:25.370761 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:25.377240 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 20:09:25.422467 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:25.656791 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:25.663058 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:09:25.728395 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:25.806468 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:25.851537 31208 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:25.853176 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:25.857127 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:25.860803 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:25.940044 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:25.953778 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:25.977241 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:25.992328 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:26.095599 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:26.238780 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:26.256562 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 896: this message will repeat every 5th retry.
W20240501 20:09:26.273221 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:26.274062 31208 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:26.340921 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:26.340981 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:26.406268 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:26.410825 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:26.433669 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:26.524500 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:26.558473 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:26.568058 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:26.614699 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:26.614789 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:26.637863 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 20:09:26.733834 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:26.737203 31208 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:26.762351 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:26.799037 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:26.801082 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:26.865191 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:26.877740 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:26.961732 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:26.997660 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:27.007093 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:27.009732 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
I20240501 20:09:27.130760 31208 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:27.212785 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:27.305428 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:27.356434 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 20:09:27.380712 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:27.394847 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:27.406687 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 20:09:27.411974 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:27.464179 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:27.547613 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:27.585937 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:27.599679 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:27.623687 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:09:27.647215 31195 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:27.678943 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:27.683871 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 20:09:27.763072 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:27.878371 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:27.978590 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 20:09:28.008278 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:28.037520 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:28.059299 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:28.066232 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:28.069473 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:28.192420 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:09:28.208181 31195 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:28.306089 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:28.437814 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:28.440615 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:28.475268 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:28.495000 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:28.497838 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:28.513279 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:28.659409 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:28.739645 31315 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:28.763276 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:28.842649 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:28.907085 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:28.907737 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 901: this message will repeat every 5th retry.
W20240501 20:09:28.948850 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:28.972249 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:28.978818 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:29.047701 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:29.063895 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:29.063935 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:29.088804 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:29.091979 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:29.129298 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:29.151706 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:29.200031 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:29.241578 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:09:29.293920 31342 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:29.310773 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:29.358248 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:29.365023 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:09:29.365695 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:29.395764 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:29.419896 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 20:09:29.427143 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:29.441720 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:29.535236 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:29.535734 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 20:09:29.684612 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:29.770264 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 20:09:29.772573 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:29.805938 31342 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:29.859056 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:29.934852 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:09:29.937657 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:29.962494 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:29.977886 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:29.989235 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:30.001578 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:30.014544 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240501 20:09:30.107888 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:30.181371 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 20:09:30.192994 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:30.229897 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:30.245088 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:30.268337 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:30.338495 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:30.362655 31342 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:30.467187 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 20:09:30.528177 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:30.602432 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:30.674424 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:30.685775 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 116 similar messages]
W20240501 20:09:30.703830 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:30.745359 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:30.757124 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:30.841696 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:30.841722 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:09:30.967868 31342 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:31.039413 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.073531 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:31.080075 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:31.122591 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
I20240501 20:09:31.165504 31348 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:09:31.165624 31348 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:09:31.166052 31348 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:09:31.167062 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:31.167142 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:31.167208 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:09:31.167326 31348 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:09:31.230022 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:31.259069 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:31.335238 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:31.358768 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 20:09:31.362176 31342 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:31.367946 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.415118 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.464582 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 906: this message will repeat every 5th retry.
W20240501 20:09:31.476661 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:31.509680 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.523754 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:31.570274 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:31.602185 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.637459 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.656770 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.676621 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.705355 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.724879 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.725683 12437 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 12a50be843fb49ee8a66c9a17d271d61 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 58f750e4d9574c9ebfd3d9e57eefd411 (127.5.139.66:36895) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "58f750e4d9574c9ebfd3d9e57eefd411" member_type: VOTER last_known_addr { host: "127.5.139.66" port: 36895 } } peers { permanent_uuid: "2ca06dad25a645bca71b09741798194a" member_type: VOTER last_known_addr { host: "127.5.139.70" port: 33933 } attrs { promote: false } } peers { permanent_uuid: "1d49e80be3d04fb99b711c62d61bac09" member_type: VOTER last_known_addr { host: "127.5.139.71" port: 46083 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: NON_VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: true } }
W20240501 20:09:31.727138 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:31.736727 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 20:09:31.784971 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:31.813160 31342 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:31.815933 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:31.842356 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:31.902444 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:31.947957 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 20:09:31.993234 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:32.111876 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:32.161329 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:32.178252 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:32.225944 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 20:09:32.299589 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:32.324399 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:09:32.335633 31342 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:32.337687 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:32.397220 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:32.423482 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:32.584962 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:32.620093 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:32.732394 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:32.815236 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:32.850437 31342 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:32.875383 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:32.891317 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 20:09:32.897022 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 20:09:32.951542 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:32.992605 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:33.023718 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:33.025030 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:33.140400 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:33.229686 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:33.262794 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:33.400091 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:33.465492 31442 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:33.478268 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:33.495607 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:33.583859 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:33.588152 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:33.603989 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:33.653293 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:33.664081 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:33.705624 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:33.708201 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:33.767060 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:33.891467 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:33.923154 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:33.993964 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:34.015081 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:34.024375 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:09:34.026849 31475 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:34.052147 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
I20240501 20:09:34.055805 31348 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:09:34.055924 31348 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:09:34.056305 31348 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:09:34.057278 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:34.057394 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:34.057456 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:09:34.057572 31348 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:09:34.083340 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 911: this message will repeat every 5th retry.
W20240501 20:09:34.103530 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:34.131786 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:34.196915 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:34.217010 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:34.237469 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:34.282897 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:34.341658 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:34.372045 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 20:09:34.372409 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 20:09:34.380997 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:34.415455 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240501 20:09:34.423096 31442 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:34.459542 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:34.499938 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:34.512502 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:34.519027 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:34.530170 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 20:09:34.581169 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:34.618228 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:34.675359 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:34.683640 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:34.727277 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:34.943445 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:09:34.995554 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:35.019785 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 20:09:35.020442 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:35.026136 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:35.028594 31442 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:35.038093 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:35.071717 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:35.131783 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:35.207432 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:35.457903 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:35.458721 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
W20240501 20:09:35.468763 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 20:09:35.499255 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:35.500999 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:35.508471 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:35.546363 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:35.560910 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:35.572005 31442 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:35.655616 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:35.689417 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 122 similar messages]
W20240501 20:09:35.756240 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:35.843092 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:35.915760 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:35.955003 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:36.021730 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.069926 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:36.079283 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.086759 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:36.102346 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:09:36.111349 31442 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:36.164858 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:36.184542 31348 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:09:36.184695 31348 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:09:36.185165 31348 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:09:36.186228 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:36.186326 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:36.186409 12921 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:09:36.186538 31348 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:09:36.195518 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.331354 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:36.368820 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.404006 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:36.405602 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:36.445497 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.455911 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:36.546221 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 916: this message will repeat every 5th retry.
I20240501 20:09:36.549191 31532 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:36.569996 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:36.579067 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.610951 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.621798 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.646263 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.658131 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.708470 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:36.792791 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:36.823089 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:36.874161 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 20:09:36.880811 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:36.897806 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
I20240501 20:09:36.980152 31532 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:36.989029 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:37.006582 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:37.034284 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:37.101838 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:37.130780 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:37.167644 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:37.178651 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:37.198246 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 20:09:37.323822 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:37.442451 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:37.474782 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:37.495772 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
I20240501 20:09:37.549858 31532 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:37.559739 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:37.569602 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:37.682875 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:37.744148 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 20:09:37.749431 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:37.871788 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:37.884603 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:37.957028 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 20:09:37.960350 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:37.985922 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:38.001103 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:09:38.056183 31572 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:38.128504 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:38.137595 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:38.203531 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:38.264863 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:38.280050 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:38.476089 31532 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:38.494796 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:38.521571 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:38.599049 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:38.626516 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:38.662214 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:38.685761 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:38.720353 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:38.802862 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:38.823989 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:38.825882 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:38.832736 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:38.886806 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:38.887689 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
I20240501 20:09:38.948726 31532 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:38.977926 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:39.039209 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:39.057626 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:39.098284 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:39.121891 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:39.133301 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 921: this message will repeat every 5th retry.
W20240501 20:09:39.154827 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:39.177618 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:39.254891 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 20:09:39.285138 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 20:09:39.313905 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:39.327874 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:39.336091 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:39.398370 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:09:39.525635 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:09:39.533178 31572 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:39.537982 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:39.545460 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:39.562973 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:39.565068 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:39.637538 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:39.639993 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:39.662302 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 20:09:39.706004 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:39.731240 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:39.845620 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:39.862231 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:39.990948 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:40.009826 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 20:09:40.020669 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:40.065259 31636 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:40.096653 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:40.106750 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 20:09:40.107340 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:40.120116 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:40.225196 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:40.343863 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:40.386018 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 20:09:40.404800 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 20:09:40.440555 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:09:40.526191 31572 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:40.546283 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:40.568087 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:40.613363 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:40.663003 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:40.694523 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:09:40.711956 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:40.793802 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:40.831032 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:40.953989 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:40.962971 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:41.034260 31572 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:41.168408 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:41.168434 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:41.207818 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.212999 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.220116 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:41.232614 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.267431 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.365208 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.370692 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.373314 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:41.420222 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:41.526280 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:41.542263 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.561455 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.610186 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:41.625620 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:41.657361 31572 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:41.658279 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:41.678218 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.722002 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 926: this message will repeat every 5th retry.
W20240501 20:09:41.736948 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.826323 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:41.828616 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.884996 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:41.964769 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 20:09:41.980909 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:41.986056 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:42.018658 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:42.040553 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:42.046296 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:42.078291 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:42.088707 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 20:09:42.090420 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:42.137014 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 20:09:42.206948 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:42.244840 31698 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:42.357374 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:42.410516 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:42.439779 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:42.468436 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:42.487573 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:42.733652 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 20:09:42.735904 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
I20240501 20:09:42.767911 31698 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:42.780673 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:42.841434 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:42.855315 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 20:09:42.920063 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:42.955633 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:42.979897 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.041302 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.081265 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.151479 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:43.168035 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.247516 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:43.274677 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:09:43.338289 31730 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:43.343210 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:43.391673 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:43.423185 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:43.517527 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.517709 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.614146 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.698701 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:43.727182 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.743690 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.763103 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:43.771376 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:43.773895 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:43.819074 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 20:09:43.830564 31698 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:43.890594 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:43.925192 31719 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:09:43.925343 31719 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:09:43.925722 31719 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:09:43.926724 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:43.926822 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:43.926918 12921 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:09:43.927093 31719 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:09:43.981386 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:43.996240 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:44.051201 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:44.135210 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:44.175736 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:44.226680 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:44.240429 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:44.325436 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 931: this message will repeat every 5th retry.
W20240501 20:09:44.350991 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 20:09:44.354385 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:44.358242 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:44.361389 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:44.395027 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
I20240501 20:09:44.412623 31730 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:44.423367 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:44.424628 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:44.434175 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:44.509642 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:44.530603 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:44.618469 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 20:09:44.619130 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:44.674511 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 20:09:44.681674 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:44.708482 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:44.771144 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:44.899694 31698 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:44.900524 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:44.928401 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:45.029314 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:09:45.030050 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:45.136483 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 20:09:45.137249 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:45.178993 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 20:09:45.181949 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:45.226163 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 20:09:45.234640 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:45.245429 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:45.253751 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:45.346738 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:45.396241 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:09:45.405742 31698 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:45.482015 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:45.495602 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:45.534499 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:45.538936 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:45.615412 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:45.699354 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:09:45.734690 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:45.787612 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:45.942644 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:45.958144 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:09:45.974385 31698 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:46.026436 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:46.034853 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:46.206337 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:46.207934 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.250317 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.308727 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.322270 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:46.400182 31801 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:46.402868 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.418648 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.428838 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:46.435729 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:46.446411 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.458616 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.581933 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.650372 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:46.661181 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:46.666257 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.752091 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:46.760715 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:46.764559 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
I20240501 20:09:46.781157 31801 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:46.806573 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 936: this message will repeat every 5th retry.
W20240501 20:09:46.837261 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:46.858901 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.863641 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 20:09:46.883337 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:46.935626 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:46.976513 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:47.049312 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:47.058833 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:47.206308 31801 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:47.220417 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:47.237313 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:47.284379 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:47.339429 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 20:09:47.383419 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:47.388375 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:47.440023 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:47.449427 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:47.469959 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:47.527994 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:47.622545 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:47.632880 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:47.736851 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:47.766671 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:47.768107 31801 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:47.785842 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:47.807473 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:47.849865 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 20:09:47.916760 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:47.939412 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 20:09:47.985432 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:48.104399 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:48.227386 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:48.229615 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:09:48.270298 31861 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:48.271719 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:48.387193 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:48.388440 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:48.461021 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:48.517648 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:48.609133 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:48.691614 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:48.706588 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:48.778544 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:48.780951 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:48.783576 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:48.800093 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 20:09:48.821498 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:48.826346 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:09:48.834939 31861 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:48.922565 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:48.940788 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:48.961568 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:48.995361 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:49.044716 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:49.055510 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 20:09:49.062620 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 20:09:49.103958 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:49.181131 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:49.295358 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:09:49.388001 31861 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:49.398514 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:49.430044 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 941: this message will repeat every 5th retry.
W20240501 20:09:49.488991 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:49.511248 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:49.544075 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:49.550765 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:49.557585 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:49.568459 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:49.591030 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:49.624142 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:49.695287 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:49.776273 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:49.779042 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:49.805570 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:49.858457 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:09:49.888800 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:49.888962 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
I20240501 20:09:49.891796 31906 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:49.933569 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:49.992293 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:50.016884 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:50.024109 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:50.044916 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 111 similar messages]
W20240501 20:09:50.147223 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:50.150632 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 20:09:50.282337 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 20:09:50.353380 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:50.362807 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:50.407558 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:50.410601 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
I20240501 20:09:50.444617 31918 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:50.498766 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:50.511798 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:50.531308 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:50.580036 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:50.616677 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:50.681701 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:50.749177 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 124 similar messages]
W20240501 20:09:50.787335 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:09:50.876457 31906 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:50.885663 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:50.945098 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:50.950626 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:50.960063 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:50.999462 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:51.017602 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:51.018838 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:51.229336 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:51.260962 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 20:09:51.284359 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:09:51.318974 31918 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:51.352193 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:51.409821 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:51.417574 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:51.418300 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:51.482898 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:51.571697 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:51.579505 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:51.609700 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:51.612376 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:51.614074 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:51.665289 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 20:09:51.804807 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:51.837718 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 946: this message will repeat every 5th retry.
W20240501 20:09:51.853250 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
I20240501 20:09:51.865639 31918 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:51.921375 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:52.004289 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:52.043745 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:52.128943 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:52.169610 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:52.199117 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:52.205533 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:52.282301 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:52.285140 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:52.302866 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:52.304996 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:52.313890 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:52.316262 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:09:52.364748 31956 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:52.470175 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 20:09:52.517518 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:52.598906 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:52.607578 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:52.637722 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:52.644881 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:52.723287 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 20:09:52.817543 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:52.866587 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:52.876617 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 20:09:52.926246 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:52.981272 31956 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:52.994555 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:53.006130 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:53.050513 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:53.088565 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:53.172024 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:53.202462 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:53.292078 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:53.323218 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:53.387014 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:53.390363 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:53.423918 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:53.530246 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:09:53.532061 31918 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:53.580437 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:53.630251 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:53.688755 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:53.690399 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:53.709764 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:53.832581 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:53.878252 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 20:09:53.899348 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:53.905202 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:53.914435 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:53.954161 31918 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:54.002415 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:54.053192 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:54.102701 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:54.163980 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:54.187304 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 20:09:54.253587 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:54.313436 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:54.336925 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:54.385011 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 951: this message will repeat every 5th retry.
W20240501 20:09:54.438681 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:54.439206 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
I20240501 20:09:54.443732 31918 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:54.535840 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:54.590788 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:54.650673 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:54.660962 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:54.689592 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:54.699641 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:54.737740 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:54.781715 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:54.844092 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:54.863307 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:54.873633 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:09:54.919642 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:09:54.935938 31956 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:54.936872 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:55.061743 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:55.066864 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 107 similar messages]
W20240501 20:09:55.083351 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 20:09:55.083757 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:55.101467 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 20:09:55.147815 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:55.151386 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240501 20:09:55.164908 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:55.222434 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:55.249163 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:55.269003 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 20:09:55.313030 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:55.496937 31956 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:55.504071 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:55.582275 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:55.596239 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:55.656399 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:55.719429 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:55.724874 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:55.727546 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:55.765936 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:55.792091 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:55.802929 12662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 120 similar messages]
W20240501 20:09:56.001646 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:09:56.021553 32052 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:56.127431 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:56.146106 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:56.162736 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:56.178670 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:56.187445 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:56.199132 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:56.217125 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:56.264902 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 20:09:56.271339 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:56.288519 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:56.300859 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:56.491235 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:56.538055 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
I20240501 20:09:56.548362 31956 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:56.574702 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:56.635390 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:56.638227 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:56.700740 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:56.795781 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:56.818361 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:56.824785 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 20:09:56.928146 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:56.966784 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 956: this message will repeat every 5th retry.
W20240501 20:09:56.983281 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:57.034783 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:57.068600 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 20:09:57.073499 32091 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:57.092691 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:57.188851 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:57.205238 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:57.209545 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:57.217945 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:57.265386 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:57.270519 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:57.392721 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:57.532200 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:57.552521 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:57.558820 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:57.579113 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 20:09:57.584897 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:57.593703 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:57.623144 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:57.624692 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:57.645788 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
I20240501 20:09:57.647990 32109 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:57.765614 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:57.932778 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 20:09:57.993930 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:58.004617 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:58.032114 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:58.063964 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:09:58.086571 32091 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:58.145824 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:58.181911 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:58.202474 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:58.246759 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:58.283319 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:58.331696 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:58.426927 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:58.571524 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:58.585884 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:58.589969 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:58.616391 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:09:58.637122 32091 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:58.698418 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:58.699369 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:58.735612 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:58.770669 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:58.796906 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 20:09:58.877343 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:58.878932 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:58.969275 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:59.026163 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:59.093092 32136 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:59.096652 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:09:59.126478 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:59.153185 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:59.184458 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:59.184540 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:59.254014 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 20:09:59.351740 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:59.442790 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:09:59.455495 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:09:59.495985 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:59.506070 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:59.601188 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:59.607198 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:09:59.614814 32136 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:09:59.619354 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:09:59.624647 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:09:59.715274 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:09:59.757709 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:59.798975 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 961: this message will repeat every 5th retry.
W20240501 20:09:59.843441 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:09:59.905705 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:09:59.936580 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:09:59.959743 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:00.028677 12924 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 20:10:00.029512 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 20:10:00.034101 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 20:10:00.102300 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 20:10:00.103142 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:10:00.121362 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
I20240501 20:10:00.139648 32176 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:00.139761 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:00.160905 12790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240501 20:10:00.191360 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:00.212767 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:00.250809 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:00.288635 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 20:10:00.451663 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:00.465946 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:00.494390 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:00.513559 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:00.628968 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:10:00.703563 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:10:00.744280 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:00.745169 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:00.760701 32136 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:00.857862 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:10:00.945400 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 20:10:00.971894 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:01.008883 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:01.035467 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:01.085908 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:10:01.099061 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 20:10:01.099975 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:01.158861 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:10:01.171183 32194 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:01.208124 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:01.231599 12439 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111) [suppressed 12 similar messages]
W20240501 20:10:01.232416 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 22b272a8582d4060a6a527ac4fce9ae5: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:10:01.305599 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:01.327646 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:01.362005 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:01.372676 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:01.379348 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:01.535789 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:01.545317 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:01.585706 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:01.623584 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:01.627097 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:10:01.683171 32223 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:01.757218 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 20:10:01.808501 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:01.848685 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:01.961218 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:02.037755 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:02.076208 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:10:02.078393 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:02.087487 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:02.102854 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:02.125586 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:02.132244 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:10:02.140255 32223 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 20:10:02.161690 32108 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
W20240501 20:10:02.161789 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:10:02.161823 32108 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:10:02.162223 32108 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:10:02.163197 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:02.163348 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:10:02.163429 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:10:02.163563 32108 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:10:02.299304 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 966: this message will repeat every 5th retry.
W20240501 20:10:02.311633 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 20:10:02.340570 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:02.447937 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:02.464769 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:02.481106 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:02.487341 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:02.503774 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:02.523177 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:02.591262 12439 catalog_manager.cc:4573] TS 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 557994658e2c4c8bbc1eeacdab958ddd: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111)
W20240501 20:10:02.595947 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 20:10:02.596975 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
I20240501 20:10:02.609273 32194 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:02.618260 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:02.744683 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:02.757328 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:02.813011 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:02.819412 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:02.874609 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:02.884729 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:02.890992 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:03.111322 32194 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:03.114996 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:03.273556 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:03.277566 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:03.315686 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:03.347848 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:03.360733 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:03.409883 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:03.534448 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:03.551609 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 20:10:03.634816 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
I20240501 20:10:03.699524 32194 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:03.701727 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:03.717135 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:03.724910 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:03.783725 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:03.811278 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:03.843551 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:03.845324 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:03.927681 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:04.125239 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:10:04.130296 32280 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:04.131899 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:04.176893 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:04.193665 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:04.206092 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:04.265177 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:04.359541 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:04.375017 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 20:10:04.411592 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:04.576400 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:10:04.579182 32280 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:04.580929 12440 catalog_manager.cc:4573] TS 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8bc9fe9ade8246779164d6db31b11e16: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111)
W20240501 20:10:04.628602 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:04.654930 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:04.660579 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:04.663020 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:04.679033 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 20:10:04.690826 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:04.696283 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:04.746388 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:04.746659 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:04.761512 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:04.816442 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:04.817562 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 971: this message will repeat every 5th retry.
W20240501 20:10:04.840340 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:04.863909 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:04.922801 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:05.057709 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 20:10:05.068173 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:05.068449 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:05.098739 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:05.115664 12531 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 108 similar messages]
W20240501 20:10:05.132836 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:05.163158 12793 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111) [suppressed 90 similar messages]
I20240501 20:10:05.201086 32327 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:05.211582 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 20:10:05.222373 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:05.300474 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:05.318451 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:05.323896 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 20:10:05.442996 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:05.466251 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:05.517733 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:05.580874 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:05.615492 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:05.805227 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
I20240501 20:10:05.822888 32344 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:05.851423 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:05.873920 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:05.934985 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:06.026355 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:06.036365 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 118 similar messages]
W20240501 20:10:06.037184 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:06.043612 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:06.096331 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:06.153168 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:10:06.200645 32344 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:06.211133 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 20:10:06.321911 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:06.346190 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:06.348209 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:06.408699 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:06.419592 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:06.472801 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:06.524380 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:06.580456 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
I20240501 20:10:06.583703 32327 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:06.701927 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:06.743383 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:06.799401 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:06.846740 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:06.861356 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:06.958494 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 20:10:07.010970 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 20:10:07.026432 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:07.058784 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:07.060045 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:07.064206 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:07.065510 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:07.077934 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:10:07.118009 32379 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:07.133558 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:07.161437 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:07.202236 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 976: this message will repeat every 5th retry.
W20240501 20:10:07.306375 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:07.315533 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:07.352056 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:07.386811 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:07.410341 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 20:10:07.444479 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:07.459369 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 20:10:07.464679 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:07.471261 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:07.474481 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:07.500241 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:07.557616 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:10:07.659654 32380 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:10:07.659791 32380 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:10:07.660200 32380 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:10:07.661213 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:07.661329 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:07.661412 12921 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:10:07.661542 32380 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 20:10:07.698881 32397 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:07.826577 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:07.849368 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:07.867326 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:07.873824 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:07.920172 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:07.949932 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:07.988966 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:07.994709 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:08.072947 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:08.182637 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:08.204857 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:08.260239 32397 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:08.386119 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:08.397842 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:08.466925 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:08.555409 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:08.580720 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:08.599709 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:08.654265 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 20:10:08.666584 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:10:08.692962 32397 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:08.715369 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:08.806711 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:08.813735 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:08.851408 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:08.955633 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:08.970465 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:08.974434 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:08.975919 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:09.070969 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:09.108247 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:09.130949 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:09.228278 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:09.291533 32379 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:09.301021 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:09.386126 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 20:10:09.389268 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:09.419669 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:09.453946 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:09.552281 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:09.554219 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:09.595917 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:09.610262 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:09.619287 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 20:10:09.629489 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:09.642522 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 981: this message will repeat every 5th retry.
W20240501 20:10:09.648886 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:09.658510 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:10:09.713904 32397 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:09.757479 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:09.767019 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:09.846462 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 20:10:09.939798 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:09.947180 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:09.948395 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:09.970348 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:09.975744 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 20:10:10.040001 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:10.072264 12923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 20:10:10.099869 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:10.112820 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:10.123369 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:10:10.131116 32397 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:10.132494 12528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 112 similar messages]
W20240501 20:10:10.133029 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:10.256457 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240501 20:10:10.315976 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:10.443835 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:10.452903 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:10.567473 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:10.591161 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:10.607596 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 20:10:10.624454 32379 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:10.633119 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:10.706936 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:10.750193 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:10.793655 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:10.862377 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:10.914700 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:10.922924 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:10.976135 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:11.025213 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:10:11.043569 32397 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:11.110846 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 123 similar messages]
W20240501 20:10:11.111479 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:11.116286 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:11.173681 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:11.203554 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 20:10:11.260947 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:11.464818 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:11.488840 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:11.505869 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:11.518815 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:11.542376 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:10:11.556272 32458 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:10:11.556429 32458 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:10:11.556874 32458 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:10:11.557991 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:11.558295 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:10:11.558367 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:10:11.558487 32496 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:10:11.569373 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:11.584517 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:11.603840 32379 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:11.612700 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:11.634750 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:11.730558 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:11.865090 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:11.865451 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:11.959662 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:11.975487 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:11.998219 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:12.026438 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:12.028343 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 20:10:12.094738 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:12.095759 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:12.128463 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:10:12.131211 32515 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:12.177971 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:12.235459 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 20:10:12.244158 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:12.296285 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:12.381055 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 20:10:12.429703 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:12.447518 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:12.507761 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 986: this message will repeat every 5th retry.
W20240501 20:10:12.537989 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:10:12.552399 32515 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:12.572643 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:12.576229 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:12.580865 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:12.652623 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:12.694213 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 20:10:12.697602 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:12.718904 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:12.748258 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:12.850850 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:13.041651 32379 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:13.046989 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:13.057536 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:13.073695 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:13.115446 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:13.123289 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:13.148363 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:13.170187 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:13.352571 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:13.452440 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:13.464215 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:13.476310 32515 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:13.496759 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 20:10:13.507462 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:13.615952 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:13.686921 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:13.719350 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:13.721758 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 20:10:13.855404 32515 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:13.928762 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:13.983448 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:14.002588 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:14.055676 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:14.068871 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:14.101069 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:14.150251 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:14.185353 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:14.242611 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:14.267040 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:10:14.327630 32515 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:14.327682 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:14.356194 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:14.368896 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:14.479013 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:14.494776 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:14.561053 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 20:10:14.573953 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:14.601884 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:14.663424 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:14.664093 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:14.677448 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 20:10:14.745107 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:14.746376 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:10:14.770179 32379 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:14.778731 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:14.814965 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:14.843740 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 991: this message will repeat every 5th retry.
W20240501 20:10:14.871122 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:14.982409 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:15.045897 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 20:10:15.080816 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:15.146660 12530 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 109 similar messages]
W20240501 20:10:15.147290 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:15.173839 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:15.184677 12921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 20:10:15.208884 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:15.222299 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:15.254202 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 20:10:15.257658 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:15.268779 12792 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240501 20:10:15.282117 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:15.305145 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:10:15.391242 32379 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:15.422240 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:15.484303 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:15.542897 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:15.585094 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:15.596063 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:15.701125 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:15.701197 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:15.723001 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:10:15.795876 32379 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:15.800259 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:15.903429 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 20:10:15.918910 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:15.958765 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:15.991811 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:16.062480 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:16.134881 12659 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111) [suppressed 117 similar messages]
W20240501 20:10:16.135564 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:16.156009 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:10:16.252009 32515 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:16.259306 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:16.375422 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:16.383836 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:16.493996 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:16.520143 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:16.566455 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:16.591476 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:16.617241 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:16.622427 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:16.672797 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:16.678925 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:10:16.794726 32515 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:16.810287 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:16.816541 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:16.821574 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:16.824470 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:16.883301 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:16.949869 12923 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 20:10:17.024134 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:17.173923 32638 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:17.195300 12528 consensus_peers.cc:485] T 7d7ea564ea504163a1e17561be9a5130 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:17.251654 12661 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 2ca06dad25a645bca71b09741798194a (127.5.139.70:33933): Couldn't send request to peer 2ca06dad25a645bca71b09741798194a. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.70:33933: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 20:10:17.256640 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:17.264214 12792 consensus_peers.cc:485] T d75bd8f836b947248ee124f456c7d721 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:17.359302 12659 consensus_peers.cc:485] T c7ddb6d79d384d39a709b5824f936d53 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:17.370126 12924 consensus_peers.cc:485] T 851eeb0ab3364529907d9d09f2a4022e P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 996: this message will repeat every 5th retry.
W20240501 20:10:17.380520 12530 consensus_peers.cc:485] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:17.425812 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:17.428201 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:17.436817 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:17.443274 12531 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:17.474071 12923 consensus_peers.cc:485] T 8bc9fe9ade8246779164d6db31b11e16 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 20:10:17.490813 12528 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:10:17.549008 32638 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:17.574064 12923 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
I20240501 20:10:17.608637 32496 raft_consensus.cc:482] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c6d2c7096e404ea9a6f73b123d74c622)
I20240501 20:10:17.608776 32496 raft_consensus.cc:504] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "9069ec6fcdb248479037f2d37e0e851d" member_type: VOTER last_known_addr { host: "127.5.139.69" port: 34719 } attrs { promote: false } } peers { permanent_uuid: "c6d2c7096e404ea9a6f73b123d74c622" member_type: VOTER last_known_addr { host: "127.5.139.72" port: 37453 } attrs { promote: false } } peers { permanent_uuid: "2d52575d9e1b40dca0fc553d7295eb74" member_type: VOTER last_known_addr { host: "127.5.139.68" port: 41997 } attrs { promote: false } }
I20240501 20:10:17.609287 32496 leader_election.cc:288] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719), c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453)
W20240501 20:10:17.610204 12921 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:17.610373 12923 leader_election.cc:334] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
I20240501 20:10:17.610447 12923 leader_election.cc:302] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2d52575d9e1b40dca0fc553d7295eb74; no voters: 9069ec6fcdb248479037f2d37e0e851d, c6d2c7096e404ea9a6f73b123d74c622
I20240501 20:10:17.610572 32496 raft_consensus.cc:2726] T 04ccc8addc55441b9ab9d8265b90b03b P 2d52575d9e1b40dca0fc553d7295eb74 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 20:10:17.680269 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:17.745568 12659 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:17.810884 12924 consensus_peers.cc:485] T 955edb7c12fa4b32ac58827f131931e3 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1001: this message will repeat every 5th retry.
W20240501 20:10:17.832935 12531 consensus_peers.cc:485] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:17.852644 12530 consensus_peers.cc:485] T 544caa38514643d5b81cd6ed85e0bae7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:17.864081 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:17.924495 12792 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:17.942812 12790 consensus_peers.cc:485] T cf6948589837482a9f580731e3b861b8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:17.947005 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:17.974381 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:18.086316 12792 consensus_peers.cc:485] T 557994658e2c4c8bbc1eeacdab958ddd P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:18.093493 12790 consensus_peers.cc:485] T 3e84a3865ce64eaebad4df46ec01842d P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
I20240501 20:10:18.119345 32638 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:18.164166 12790 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:18.278267 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:18.281093 12921 consensus_peers.cc:485] T f08535cb2f46464b9e510a45fee67b4f P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:18.444860 12659 consensus_peers.cc:485] T 61bc22cdc6ab45529066fe6f4d14d4c7 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:18.474133 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:18.500548 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:18.536808 12659 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:18.545670 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
I20240501 20:10:18.547111 32638 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:18.600622 12662 consensus_peers.cc:485] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 1006: this message will repeat every 5th retry.
W20240501 20:10:18.635457 12659 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:18.721383 12662 consensus_peers.cc:485] T 800d5f43cd1b4fb6bd8bbcac262aeb80 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:18.798559 12661 consensus_peers.cc:485] T 745e39ec23db4f90bb8b1e8b49a50244 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:18.857555 12661 consensus_peers.cc:462] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111)
W20240501 20:10:18.861860 12662 consensus_peers.cc:485] T 8b61e94fc7e541c8aa84957560508a31 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 20:10:18.893365 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:18.932619 12792 consensus_peers.cc:485] T 0fb481d394034a8f85af613beb5c9ec8 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:18.977203 12530 consensus_peers.cc:485] T d940b16d07d54945abe4a10a11dd6a2a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
I20240501 20:10:18.991945 32670 raft_consensus.cc:1013] T 12a50be843fb49ee8a66c9a17d271d61 P 58f750e4d9574c9ebfd3d9e57eefd411 [term 1 LEADER]: attempt to promote peer 2d52575d9e1b40dca0fc553d7295eb74: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
W20240501 20:10:19.034233 12662 consensus_peers.cc:485] T 32a17c2e392a49b7b181fa7a25a778a1 P 58f750e4d9574c9ebfd3d9e57eefd411 -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:19.072758 12921 consensus_peers.cc:485] T 706878734216445f8422e422228d4230 P 2d52575d9e1b40dca0fc553d7295eb74 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Couldn't send request to peer 9069ec6fcdb248479037f2d37e0e851d. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 20:10:19.084333 12528 consensus_peers.cc:462] T 7e4d79f5e11b4923acb2ce543884b54a P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:19.139825 12790 consensus_peers.cc:462] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:19.222852 12530 consensus_peers.cc:485] T ce6add2f19a649b3a12497951047705c P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn't send request to peer c6d2c7096e404ea9a6f73b123d74c622. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.72:37453: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:19.238662 12793 consensus_peers.cc:485] T 0bf64d9f3ce94db28b032cdcd795a255 P a238fc0c6d844cedb8adccb97d01ae5e -> Peer 1d49e80be3d04fb99b711c62d61bac09 (127.5.139.71:46083): Couldn't send request to peer 1d49e80be3d04fb99b711c62d61bac09. Status: Network error: Client connection negotiation failed: client connection to 127.5.139.71:46083: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 20:10:19.325030 12528 consensus_peers.cc:462] T 22b272a8582d4060a6a527ac4fce9ae5 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer 9069ec6fcdb248479037f2d37e0e851d (127.5.139.69:34719): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.5.139.69:34719: connect: Connection refused (error 111)
W20240501 20:10:19.350375 12530 consensus_peers.cc:485] T db240ec5f8b2444fb6114ad814f3cdb7 P ebbbd9ac07a94e8dbbae6def0a2df415 -> Peer c6d2c7096e404ea9a6f73b123d74c622 (127.5.139.72:37453): Couldn'